Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 891084 Details for
Bug 1092914
RHEV host turns unassigned for 5 minutes in UI in case NFS mount error happens during upgrade
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
engine log
engine.log (text/plain), 9.88 MB, created by
Tomas Dosek
on 2014-04-30 09:01:18 UTC
(
hide
)
Description:
engine log
Filename:
MIME Type:
Creator:
Tomas Dosek
Created:
2014-04-30 09:01:18 UTC
Size:
9.88 MB
patch
obsolete
>2014-03-07 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-79) Start deleteAgedOutAuditLogs >2014-03-07 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-79) Finished deleteAgedOutAuditLogs >2014-03-07 03:55:00,344 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-03-07 03:55:00,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-03-07 04:55:00,346 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-03-07 04:55:00,925 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-03-07 05:55:00,348 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-03-07 05:55:00,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-03-07 06:55:00,349 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-30) Start refreshing all users data >2014-03-07 06:55:00,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-03-07 07:55:00,350 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-03-07 07:55:00,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-03-07 08:55:00,352 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-03-07 08:55:00,929 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-03-07 09:55:00,353 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-03-07 09:55:00,931 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-03-07 10:55:00,354 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-03-07 10:55:00,931 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-03-07 11:55:00,356 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-03-07 11:55:00,932 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-03-07 12:55:00,358 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) Start refreshing all users data >2014-03-07 12:55:00,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-03-07 13:55:00,359 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-03-07 13:55:00,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-03-07 14:55:00,361 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-07 14:55:00,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-03-07 15:06:15,731 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-25) Failed to invoke scheduled method refreshLightWeightData: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:147) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:112) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:96) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:89) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterSyncJob.refreshLightWeightData(GlusterSyncJob.java:78) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139) > ... 21 more >Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:377) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329) > ... 24 more >Caused by: javax.resource.ResourceException: Could not create connection > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:288) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:246) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:781) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:344) > ... 27 more >Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) > at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:123) > at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:28) > at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:20) > at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30) > at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:22) > at org.postgresql.Driver.makeConnection(Driver.java:391) > at org.postgresql.Driver.connect(Driver.java:265) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:260) > ... 30 more >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:528) [rt.jar:1.7.0_51] > at java.net.Socket.<init>(Socket.java:425) [rt.jar:1.7.0_51] > at java.net.Socket.<init>(Socket.java:208) [rt.jar:1.7.0_51] > at org.postgresql.core.PGStream.<init>(PGStream.java:60) > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:74) > ... 39 more > >2014-03-07 15:06:15,729 ERROR [org.ovirt.engine.core.utils.timer.SchedulerUtilQuartzImpl] (DefaultQuartzScheduler_Worker-11) Failed to invoke scheduled method refreshGlusterServices: java.lang.reflect.InvocationTargetException > at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) [scheduler.jar:] > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) [quartz.jar:] > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) [quartz.jar:] >Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: Could not get JDBC Connection; nested exception is java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:80) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:573) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:637) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:666) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:706) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.executeCallInternal(PostgresDbEngineDialect.java:154) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.PostgresDbEngineDialect$PostgresSimpleJdbcCall.doExecute(PostgresDbEngineDialect.java:120) [dal.jar:] > at org.springframework.jdbc.core.simple.SimpleJdbcCall.execute(SimpleJdbcCall.java:181) [spring-jdbc.jar:3.1.1.RELEASE] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeImpl(SimpleJdbcCallsHandler.java:147) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadAndReturnMap(SimpleJdbcCallsHandler.java:112) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.SimpleJdbcCallsHandler.executeReadList(SimpleJdbcCallsHandler.java:103) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:96) [dal.jar:] > at org.ovirt.engine.core.dao.VdsGroupDAODbFacadeImpl.getAll(VdsGroupDAODbFacadeImpl.java:89) [dal.jar:] > at org.ovirt.engine.core.bll.gluster.GlusterServiceSyncJob.refreshGlusterServices(GlusterServiceSyncJob.java:54) [bll.jar:] > ... 6 more >Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:147) > at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:111) [spring-jdbc.jar:3.1.1.RELEASE] > at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:77) [spring-jdbc.jar:3.1.1.RELEASE] > ... 19 more >Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/ENGINEDataSource > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:390) > at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464) > at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139) > ... 21 more >Caused by: javax.resource.ResourceException: IJ000658: Unexpected throwable while trying to create a connection: null > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:377) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:397) > at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:365) > at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329) > ... 24 more >Caused by: javax.resource.ResourceException: Could not create connection > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:288) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(LocalManagedConnectionFactory.java:246) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:781) > at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:344) > ... 27 more >Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:136) > at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) > at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:123) > at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:28) > at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:20) > at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:30) > at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:22) > at org.postgresql.Driver.makeConnection(Driver.java:391) > at org.postgresql.Driver.connect(Driver.java:265) > at org.jboss.jca.adapters.jdbc.local.LocalManagedConnectionFactory.getLocalManagedConnection(LocalManagedConnectionFactory.java:260) > ... 30 more >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) [rt.jar:1.7.0_51] > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) [rt.jar:1.7.0_51] > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:579) [rt.jar:1.7.0_51] > at java.net.Socket.connect(Socket.java:528) [rt.jar:1.7.0_51] > at java.net.Socket.<init>(Socket.java:425) [rt.jar:1.7.0_51] > at java.net.Socket.<init>(Socket.java:208) [rt.jar:1.7.0_51] > at org.postgresql.core.PGStream.<init>(PGStream.java:60) > at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:74) > ... 39 more > >2014-03-07 15:18:19,053 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-03-07 15:18:19,071 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf". >2014-03-07 15:18:19,071 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-03-07 15:18:19,071 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-03-07 15:18:19,072 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-03-07 15:18:19,072 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-03-07 15:18:19,072 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-03-07 15:18:19,072 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-03-07 15:18:19,072 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-03-07 15:18:19,072 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_AJP_PORT" is "8702". >2014-03-07 15:18:19,072 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_APPS" is "engine.ear redhat-support-plugin-rhev.ear". >2014-03-07 15:18:19,072 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-03-07 15:18:19,072 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-03-07 15:18:19,072 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-03-07 15:18:19,073 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-03-07 15:18:19,073 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-03-07 15:18:19,073 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_HOST" is "localhost". >2014-03-07 15:18:19,073 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_PORT" is "5432". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SECURED" is "False". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_SSL" is "false". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DB_USER" is "engine". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_FQDN" is "localhost.localdomain". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_GROUP" is "ovirt". >2014-03-07 15:18:19,076 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-03-07 15:18:19,077 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-03-07 15:18:19,077 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-03-07 15:18:19,077 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-03-07 15:18:19,077 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_HTTP_PORT" is "None". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PERM_MAX" is "256m". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PERM_MIN" is "256m". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-03-07 15:18:19,079 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_STOP_TIME" is "10". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-03-07 15:18:19,080 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_USER" is "ovirt". >2014-03-07 15:18:19,082 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-03-07 15:18:19,082 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-03-07 15:18:19,082 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-03-07 15:18:19,082 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-03-07 15:18:19,082 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 39) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-03-07 15:18:20,459 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 39) Start initializing Backend >2014-03-07 15:18:20,840 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ServerService Thread Pool -- 39) Failed to decrypt value for property LocalAdminPassword will be used encrypted value: javax.crypto.BadPaddingException: Data must start with zero > at sun.security.rsa.RSAPadding.unpadV15(RSAPadding.java:329) [rt.jar:1.7.0_51] > at sun.security.rsa.RSAPadding.unpad(RSAPadding.java:272) [rt.jar:1.7.0_51] > at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:356) [sunjce_provider.jar:1.7.0_51] > at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:382) [sunjce_provider.jar:1.7.0_51] > at javax.crypto.Cipher.doFinal(Cipher.java:1922) [jce.jar:1.7.0_51] > at org.ovirt.engine.core.utils.crypt.EngineEncryptionUtils.decrypt(EngineEncryptionUtils.java:169) [utils.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.getValue(DBConfigUtils.java:107) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.updateOption(DBConfigUtils.java:275) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.refreshVdcOptionCache(DBConfigUtils.java:35) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.<init>(DBConfigUtils.java:166) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.<init>(DBConfigUtils.java:152) [dal.jar:] > at org.ovirt.engine.core.bll.Backend.Initialize(Backend.java:183) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.create(Backend.java:130) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:130) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:125) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:140) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > at org.jboss.threads.JBossThread.run(JBossThread.java:122) > >2014-03-07 15:18:20,961 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 39) Could not find enum value for option: WebAdminUpperRightButtonURL >2014-03-07 15:18:21,043 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 39) Running ovirt-engine 3.3.1-2.BZ1070265.el6ev >2014-03-07 15:18:21,048 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 39) Start initializing dictionaries >2014-03-07 15:18:21,058 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 39) Finished initializing dictionaries >2014-03-07 15:18:21,058 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 39) Start initializing AuditLogCleanupManager >2014-03-07 15:18:21,059 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 39) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-03-07 15:18:21,088 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 39) Finished initializing AuditLogCleanupManager >2014-03-07 15:18:21,093 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 39) Start initializing TagsDirector >2014-03-07 15:18:21,108 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 39) Tag root added to tree >2014-03-07 15:18:21,127 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 39) Finished initializing TagsDirector >2014-03-07 15:18:21,129 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 39) Start initializing IsoDomainListSyncronizer >2014-03-07 15:18:21,140 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 39) Finished initializing IsoDomainListSyncronizer >2014-03-07 15:18:21,157 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 39) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-03-07 15:18:21,385 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 39) Completed initializing handlers >2014-03-07 15:18:21,387 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 39) Start initializing ErrorTranslatorImpl >2014-03-07 15:18:21,413 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 39) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-03-07 15:18:21,414 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 39) Finished initializing ErrorTranslatorImpl >2014-03-07 15:18:21,414 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 39) Start initializing ErrorTranslatorImpl >2014-03-07 15:18:21,414 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 39) Finished initializing ErrorTranslatorImpl >2014-03-07 15:18:21,415 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 39) Mark incomplete jobs as UNKNOWN >2014-03-07 15:18:21,479 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 39) Start initializing JobRepositoryCleanupManager >2014-03-07 15:18:21,479 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 39) Finished initializing JobRepositoryCleanupManager >2014-03-07 15:18:21,480 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 39) Start initializing AutoRecoveryManager >2014-03-07 15:18:21,481 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 39) Finished initializing AutoRecoveryManager >2014-03-07 15:18:21,482 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 39) Start initializing ExecutionMessageDirector >2014-03-07 15:18:21,489 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 39) Finished initializing ExecutionMessageDirector >2014-03-07 15:18:21,521 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 36) Start initializing UsersDomainsCacheManagerService >2014-03-07 15:18:21,541 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (ServerService Thread Pool -- 36) Start initializing DbUserCacheManager >2014-03-07 15:18:21,546 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (ServerService Thread Pool -- 36) Finished initializing DbUserCacheManager >2014-03-07 15:18:21,546 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 36) Finished initializing UsersDomainsCacheManagerService >2014-03-07 15:18:21,559 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ServerService Thread Pool -- 43) Initialization of AsyncTaskManager completed successfully. >2014-03-07 15:18:21,561 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 43) Start initializing ResourceManager >2014-03-07 15:18:21,583 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-03-07 15:18:21,797 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 43) Entered VdsManager constructor >2014-03-07 15:18:21,852 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 43) Initialize vdsBroker (10.34.27.209,54,321) >2014-03-07 15:18:22,115 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 43) VDS 53fd133a-9dc6-11e2-abd9-001a4a223991 was added to the Resource Manager >2014-03-07 15:18:22,157 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 43) Finished initializing ResourceManager >2014-03-07 15:18:22,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 43) Initialization of OvfDataUpdater completed successfully. >2014-03-07 15:18:22,170 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 43) Start scheduling to enable vds load balancer >2014-03-07 15:18:22,171 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 43) Finished scheduling to enable vds load balancer >2014-03-07 15:18:22,192 INFO [org.ovirt.engine.core.bll.network.MacPoolManager] (pool-4-thread-1) MacPoolManager(65b5e9a1): Start initializing >2014-03-07 15:18:22,226 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 43) Init VM custom properties utilities >2014-03-07 15:18:22,228 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 43) Init device custom properties utilities >2014-03-07 15:18:22,238 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 43) Initializing Scheduling manager >2014-03-07 15:18:22,271 INFO [org.ovirt.engine.core.bll.network.MacPoolManager] (pool-4-thread-1) MacPoolManager(65b5e9a1): Finished initializing >2014-03-07 15:18:22,320 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 43) External scheduler disabled, discovery skipped >2014-03-07 15:18:22,321 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 43) Initialized Scheduling manager >2014-03-07 15:18:22,321 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 43) Initializing DWH Heart Beat >2014-03-07 15:18:25,150 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Initializing Host: 10.34.27.209 >2014-03-07 15:19:15,338 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-4) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-03-07 15:19:15,349 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-4) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-03-07 15:19:24,879 ERROR [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-2) USER_FAILED_TO_AUTHENTICATE : administrator >2014-03-07 15:19:24,883 WARN [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-2) CanDoAction of action LoginAdminUser failed. Reasons:USER_FAILED_TO_AUTHENTICATE >2014-03-07 15:19:34,795 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginAdminUserCommand internal: false. >2014-03-07 15:19:34,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: UNASSIGNED not exist in string table >2014-03-07 15:19:34,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-03-07 15:19:34,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-03-07 15:19:34,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-03-07 15:19:34,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-03-07 15:19:34,877 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: DWH_STOPPED not exist in string table >2014-03-07 15:19:34,877 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: DWH_STARTED not exist in string table >2014-03-07 15:19:34,877 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: DWH_ERROR not exist in string table >2014-03-07 15:19:34,887 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: USER_FAILED_REMOVE_VM not have severity. Assumed Normal >2014-03-07 15:19:34,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: USER_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2014-03-07 15:19:34,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: USER_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2014-03-07 15:19:34,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: USER_FAILED_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2014-03-07 15:19:34,888 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not have severity. Assumed Normal >2014-03-07 15:19:34,891 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: GLUSTER_VOLUME_OPTION_CHANGED_FROM_CLI not have severity. Assumed Normal >2014-03-07 15:19:34,891 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: GLUSTER_SERVICES_LIST_NOT_FETCHED not have severity. Assumed Normal >2014-03-07 15:19:34,891 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: GLUSTER_VOLUME_BRICK_ADDED not have severity. Assumed Normal >2014-03-07 15:19:34,891 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: USER_EXTEND_DISK_SIZE_UPDATE_VM_FAILURE not have severity. Assumed Normal >2014-03-07 15:19:34,891 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: VM_MIGRATION_START_SYSTEM_INITIATED not have severity. Assumed Normal >2014-03-07 15:19:34,891 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not have severity. Assumed Normal >2014-03-07 15:19:34,891 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_UP not have severity. Assumed Normal >2014-03-07 15:19:34,892 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_DOWN not have severity. Assumed Normal >2014-03-07 15:19:34,892 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: USER_UPDATED_MOM_POLICIES not have severity. Assumed Normal >2014-03-07 15:19:34,895 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) AuditLogType: USER_FAILED_TO_UPDATE_MOM_POLICIES not have severity. Assumed Normal >2014-03-07 15:19:34,944 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-03-07 15:19:36,433 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-03-07 15:19:36,462 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-03-07 16:18:21,614 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-03-07 16:18:22,164 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-03-07 17:18:21,620 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-50) Start refreshing all users data >2014-03-07 17:18:22,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-03-07 18:18:21,622 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-03-07 18:18:22,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-03-07 19:18:21,625 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-18) Start refreshing all users data >2014-03-07 19:18:22,175 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-03-07 20:18:21,626 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-03-07 20:18:22,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-03-07 21:18:21,631 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-03-07 21:18:22,178 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-03-07 22:18:21,632 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-03-07 22:18:22,179 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-03-07 23:18:21,633 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-03-07 23:18:22,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-03-08 00:18:21,637 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-03-08 00:18:22,181 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-03-08 01:18:21,637 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-03-08 01:18:22,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-03-08 02:18:21,639 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-03-08 02:18:22,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-03-08 03:18:21,641 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-08 03:18:22,185 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-03-08 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-91) Start deleteAgedOutAuditLogs >2014-03-08 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-91) Finished deleteAgedOutAuditLogs >2014-03-08 04:18:21,642 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-18) Start refreshing all users data >2014-03-08 04:18:22,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-03-08 05:18:21,644 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-03-08 05:18:22,187 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-08 06:18:21,646 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-08 06:18:22,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-03-08 07:18:21,647 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-03-08 07:18:22,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-08 08:18:21,649 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-41) Start refreshing all users data >2014-03-08 08:18:22,193 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-03-08 09:18:21,651 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-08 09:18:22,194 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-03-08 10:18:21,653 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-86) Start refreshing all users data >2014-03-08 10:18:22,195 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-03-08 11:18:21,655 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-03-08 11:18:22,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-03-08 12:18:21,658 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-08 12:18:22,197 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-03-08 13:18:21,659 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-03-08 13:18:22,198 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-03-08 14:18:21,660 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-03-08 14:18:22,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-03-08 15:18:21,661 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-03-08 15:18:22,200 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-08 16:18:21,662 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-03-08 16:18:22,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-08 17:18:21,664 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-03-08 17:18:22,202 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-03-08 18:18:21,667 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-03-08 18:18:22,203 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-03-08 19:18:21,673 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-03-08 19:18:22,207 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-03-08 20:18:21,675 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-03-08 20:18:22,208 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-03-08 21:18:21,676 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-03-08 21:18:22,213 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-03-08 22:18:21,678 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-03-08 22:18:22,215 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-08 23:18:21,680 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-03-08 23:18:22,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-03-09 00:18:21,681 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-03-09 00:18:22,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-03-09 01:18:21,684 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-03-09 01:18:22,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-03-09 02:18:21,686 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-03-09 02:18:22,219 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-03-09 03:18:21,688 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-42) Start refreshing all users data >2014-03-09 03:18:22,220 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-03-09 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-27) Start deleteAgedOutAuditLogs >2014-03-09 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-27) Finished deleteAgedOutAuditLogs >2014-03-09 04:18:21,690 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-03-09 04:18:22,221 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-03-09 05:18:21,692 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-03-09 05:18:22,222 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-03-09 06:18:21,695 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-03-09 06:18:22,224 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-03-09 07:18:21,697 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-03-09 07:18:22,225 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-03-09 08:18:21,698 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-50) Start refreshing all users data >2014-03-09 08:18:22,226 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-03-09 09:18:21,700 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-03-09 09:18:22,227 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-03-09 10:18:21,705 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-03-09 10:18:22,229 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-03-09 11:18:21,707 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-03-09 11:18:22,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-03-09 12:18:21,709 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-03-09 12:18:22,231 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-03-09 13:18:21,713 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-03-09 13:18:22,232 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-03-09 14:18:21,714 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-03-09 14:18:22,233 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-03-09 15:18:21,716 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-03-09 15:18:22,235 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-03-09 16:18:21,718 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-03-09 16:18:22,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-03-09 17:18:21,719 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-50) Start refreshing all users data >2014-03-09 17:18:22,239 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-03-09 18:18:21,721 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-03-09 18:18:22,241 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-09 19:18:21,722 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-03-09 19:18:22,241 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-03-09 20:18:21,723 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-03-09 20:18:22,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-09 21:18:21,725 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-03-09 21:18:22,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-03-09 22:18:21,727 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-09 22:18:22,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-03-09 23:18:21,728 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) Start refreshing all users data >2014-03-09 23:18:22,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-03-10 00:18:21,729 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-03-10 00:18:22,247 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-03-10 01:18:21,731 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-03-10 01:18:22,246 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-03-10 02:18:21,733 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-03-10 02:18:22,247 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-03-10 03:18:21,734 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-03-10 03:18:22,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-03-10 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-28) Start deleteAgedOutAuditLogs >2014-03-10 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-28) Finished deleteAgedOutAuditLogs >2014-03-10 04:18:21,736 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-10 04:18:22,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-03-10 05:18:21,738 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-03-10 05:18:22,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-03-10 06:18:21,739 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-03-10 06:18:22,253 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-03-10 07:18:21,741 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-03-10 07:18:22,253 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-03-10 08:18:21,743 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-03-10 08:18:22,255 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-03-10 09:18:21,745 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-76) Start refreshing all users data >2014-03-10 09:18:22,256 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-03-10 10:18:21,748 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-03-10 10:18:22,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-03-10 11:18:21,750 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-03-10 11:18:22,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-03-10 12:18:21,751 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-03-10 12:18:22,259 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-03-10 13:18:21,753 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-03-10 13:18:22,259 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-03-10 14:18:21,755 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-03-10 14:18:22,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-03-10 15:18:21,756 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-67) Start refreshing all users data >2014-03-10 15:18:22,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-10 16:18:21,758 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-03-10 16:18:22,263 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-03-10 17:18:21,759 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-89) Start refreshing all users data >2014-03-10 17:18:22,264 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-03-10 18:18:21,761 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-03-10 18:18:22,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-03-10 19:18:21,763 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-03-10 19:18:22,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-03-10 20:18:21,765 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-03-10 20:18:22,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-03-10 21:18:21,766 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-88) Start refreshing all users data >2014-03-10 21:18:22,273 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-03-10 22:18:21,768 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-03-10 22:18:22,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-03-10 23:18:21,771 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-03-10 23:18:22,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-11 00:18:21,773 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) Start refreshing all users data >2014-03-11 00:18:22,276 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-03-11 01:18:21,774 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-03-11 01:18:22,276 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-03-11 02:18:21,776 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-54) Start refreshing all users data >2014-03-11 02:18:22,276 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-03-11 03:18:21,777 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-03-11 03:18:22,278 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-03-11 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-88) Start deleteAgedOutAuditLogs >2014-03-11 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-88) Finished deleteAgedOutAuditLogs >2014-03-11 04:18:21,779 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-03-11 04:18:22,281 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-03-11 05:18:21,780 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-03-11 05:18:22,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-03-11 06:18:21,781 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-03-11 06:18:22,284 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-03-11 07:18:21,783 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-69) Start refreshing all users data >2014-03-11 07:18:22,285 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-03-11 08:18:21,785 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-03-11 08:18:22,286 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-03-11 09:18:21,787 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-03-11 09:18:22,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-03-11 10:18:21,789 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-03-11 10:18:22,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-03-11 11:18:21,792 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-03-11 11:18:22,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-03-11 12:18:21,793 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-03-11 12:18:22,296 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-03-11 13:18:21,795 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-03-11 13:18:22,296 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-03-11 14:18:21,797 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-03-11 14:18:22,298 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-03-11 15:18:21,799 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-03-11 15:18:22,299 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-11 16:18:21,801 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-89) Start refreshing all users data >2014-03-11 16:18:22,302 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-03-11 17:18:21,803 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-03-11 17:18:22,302 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-03-11 18:18:21,804 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-03-11 18:18:22,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-03-11 19:18:21,806 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-03-11 19:18:22,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-03-11 20:18:21,807 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-03-11 20:18:22,304 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-03-11 21:18:21,809 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-87) Start refreshing all users data >2014-03-11 21:18:22,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-03-11 22:18:21,811 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-88) Start refreshing all users data >2014-03-11 22:18:22,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-03-11 23:18:21,812 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-90) Start refreshing all users data >2014-03-11 23:18:22,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-03-12 00:18:21,814 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-03-12 00:18:22,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-03-12 01:18:21,816 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-88) Start refreshing all users data >2014-03-12 01:18:22,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-03-12 02:18:21,817 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-03-12 02:18:22,307 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-03-12 03:18:21,819 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-03-12 03:18:22,307 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-12 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-37) Start deleteAgedOutAuditLogs >2014-03-12 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-37) Finished deleteAgedOutAuditLogs >2014-03-12 04:18:21,820 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-03-12 04:18:22,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-03-12 05:18:21,822 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-67) Start refreshing all users data >2014-03-12 05:18:22,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-03-12 06:18:21,824 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-12 06:18:22,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-03-12 07:18:21,825 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-50) Start refreshing all users data >2014-03-12 07:18:22,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-03-12 08:18:21,827 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-03-12 08:18:22,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-03-12 09:18:21,829 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-03-12 09:18:22,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-12 10:18:21,830 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-03-12 10:18:22,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-03-12 11:18:21,831 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-03-12 11:18:22,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-03-12 12:18:21,834 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-03-12 12:18:22,314 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-03-12 13:18:21,835 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-03-12 13:18:22,315 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-03-12 14:18:21,836 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-03-12 14:18:22,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-03-12 15:18:21,838 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-90) Start refreshing all users data >2014-03-12 15:18:22,317 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-03-12 16:18:21,840 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-87) Start refreshing all users data >2014-03-12 16:18:22,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-12 17:18:21,843 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-81) Start refreshing all users data >2014-03-12 17:18:22,319 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-03-12 18:18:21,845 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-03-12 18:18:22,321 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-03-12 19:18:21,847 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-03-12 19:18:22,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-03-12 20:18:21,849 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-88) Start refreshing all users data >2014-03-12 20:18:22,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-03-12 21:18:21,851 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-03-12 21:18:22,324 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-03-12 22:18:21,853 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-03-12 22:18:22,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-03-12 23:18:21,855 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-03-12 23:18:22,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-03-13 00:18:21,857 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-03-13 00:18:22,326 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-03-13 01:18:21,859 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-03-13 01:18:22,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-03-13 02:18:21,861 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-03-13 02:18:22,329 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-13 03:18:21,863 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-03-13 03:18:22,330 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-03-13 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-62) Start deleteAgedOutAuditLogs >2014-03-13 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-62) Finished deleteAgedOutAuditLogs >2014-03-13 04:18:21,864 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) Start refreshing all users data >2014-03-13 04:18:22,331 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-03-13 05:18:21,866 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-03-13 05:18:22,333 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-13 06:18:21,868 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-03-13 06:18:22,333 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-03-13 07:18:21,870 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-87) Start refreshing all users data >2014-03-13 07:18:22,335 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-03-13 08:18:21,872 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-03-13 08:18:22,335 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-03-13 09:18:21,874 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-03-13 09:18:22,336 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-03-13 10:18:21,875 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-03-13 10:18:22,337 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-03-13 11:18:21,877 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-25) Start refreshing all users data >2014-03-13 11:18:22,337 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-03-13 12:18:21,878 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) Start refreshing all users data >2014-03-13 12:18:22,340 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-03-13 13:18:21,879 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-03-13 13:18:22,341 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-03-13 14:18:21,881 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-41) Start refreshing all users data >2014-03-13 14:18:22,342 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-13 15:18:21,883 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-13 15:18:22,344 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-03-13 16:18:21,885 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-13 16:18:22,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-03-13 17:18:21,886 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-03-13 17:18:22,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-03-13 18:18:21,888 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-03-13 18:18:22,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-03-13 19:18:21,890 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-03-13 19:18:22,348 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-03-13 20:18:21,892 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-03-13 20:18:22,348 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-03-13 21:18:21,894 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-13 21:18:22,349 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-03-13 22:18:21,896 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-03-13 22:18:22,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-03-13 23:18:21,898 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-20) Start refreshing all users data >2014-03-13 23:18:22,351 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-14 00:18:21,899 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-03-14 00:18:22,355 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-03-14 01:18:21,900 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-62) Start refreshing all users data >2014-03-14 01:18:22,356 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-03-14 02:18:21,902 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-03-14 02:18:22,356 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-03-14 03:18:21,904 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-03-14 03:18:22,357 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-03-14 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-29) Start deleteAgedOutAuditLogs >2014-03-14 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-29) Finished deleteAgedOutAuditLogs >2014-03-14 04:18:21,906 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-03-14 04:18:22,357 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-03-14 05:18:21,907 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-03-14 05:18:22,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-03-14 06:18:21,909 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-03-14 06:18:22,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-03-14 07:18:21,911 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-92) Start refreshing all users data >2014-03-14 07:18:22,359 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-14 08:18:21,913 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-76) Start refreshing all users data >2014-03-14 08:18:22,360 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-03-14 09:18:21,915 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-03-14 09:18:22,361 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-03-14 10:18:21,917 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-03-14 10:18:22,362 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-03-14 11:18:21,919 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-14 11:18:22,363 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-03-14 12:18:21,920 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-03-14 12:18:22,364 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-03-14 13:18:21,922 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-63) Start refreshing all users data >2014-03-14 13:18:22,365 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-03-14 14:18:21,924 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-2) Start refreshing all users data >2014-03-14 14:18:22,366 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-14 15:18:21,926 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-03-14 15:18:22,367 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-03-14 16:18:21,928 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-03-14 16:18:22,368 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-03-14 17:18:21,930 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-03-14 17:18:22,369 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-03-14 18:18:21,932 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-03-14 18:18:22,370 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-03-14 19:18:21,934 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-03-14 19:18:22,372 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-03-14 20:18:21,938 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-92) Start refreshing all users data >2014-03-14 20:18:22,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-03-14 21:18:21,939 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-03-14 21:18:22,375 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-03-14 22:18:21,941 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-03-14 22:18:22,376 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-03-14 23:18:21,943 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-03-14 23:18:22,378 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-03-15 00:18:21,945 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-30) Start refreshing all users data >2014-03-15 00:18:22,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-03-15 01:18:21,946 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-03-15 01:18:22,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-03-15 02:18:21,948 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-03-15 02:18:22,380 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-03-15 03:18:21,949 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-76) Start refreshing all users data >2014-03-15 03:18:22,381 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-03-15 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-48) Start deleteAgedOutAuditLogs >2014-03-15 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-48) Finished deleteAgedOutAuditLogs >2014-03-15 04:18:21,950 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-03-15 04:18:22,382 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-03-15 05:18:21,952 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-03-15 05:18:22,383 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-03-15 06:18:21,956 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-15 06:18:22,384 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-03-15 07:18:21,958 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-03-15 07:18:22,385 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-03-15 08:18:21,959 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-03-15 08:18:22,386 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-03-15 09:18:21,961 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-2) Start refreshing all users data >2014-03-15 09:18:22,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-03-15 10:18:21,963 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-03-15 10:18:22,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-03-15 11:18:21,965 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-03-15 11:18:22,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-03-15 12:18:21,967 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-03-15 12:18:22,388 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-03-15 13:18:21,969 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-03-15 13:18:22,389 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-03-15 14:18:21,970 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-03-15 14:18:22,391 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-03-15 15:18:21,971 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-03-15 15:18:22,391 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-03-15 16:18:21,973 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-03-15 16:18:22,392 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-03-15 17:18:21,975 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-86) Start refreshing all users data >2014-03-15 17:18:22,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-03-15 18:18:21,977 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-03-15 18:18:22,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-03-15 19:18:21,978 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-41) Start refreshing all users data >2014-03-15 19:18:22,396 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-03-15 20:18:21,980 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-03-15 20:18:22,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-03-15 21:18:21,981 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-62) Start refreshing all users data >2014-03-15 21:18:22,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-03-15 22:18:21,983 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-30) Start refreshing all users data >2014-03-15 22:18:22,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-03-15 23:18:21,984 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-03-15 23:18:22,398 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-03-16 00:18:21,986 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) Start refreshing all users data >2014-03-16 00:18:22,400 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-03-16 01:18:21,987 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-03-16 01:18:22,401 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-03-16 02:18:21,989 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-03-16 02:18:22,401 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-03-16 03:18:21,991 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) Start refreshing all users data >2014-03-16 03:18:22,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-03-16 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-19) Start deleteAgedOutAuditLogs >2014-03-16 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-19) Finished deleteAgedOutAuditLogs >2014-03-16 04:18:21,993 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-16 04:18:22,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-03-16 05:18:21,995 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-6) Start refreshing all users data >2014-03-16 05:18:22,404 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-03-16 06:18:21,996 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-03-16 06:18:22,405 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-03-16 07:18:21,997 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-16 07:18:22,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-03-16 08:18:21,999 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-03-16 08:18:22,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-03-16 09:18:22,001 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-03-16 09:18:22,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-03-16 10:18:22,003 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-03-16 10:18:22,408 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-03-16 11:18:22,004 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-03-16 11:18:22,408 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-03-16 12:18:22,005 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-03-16 12:18:22,408 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-16 13:18:22,007 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-25) Start refreshing all users data >2014-03-16 13:18:22,409 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-03-16 14:18:22,009 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-54) Start refreshing all users data >2014-03-16 14:18:22,410 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-03-16 15:18:22,011 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-03-16 15:18:22,411 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-16 16:18:22,012 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-30) Start refreshing all users data >2014-03-16 16:18:22,412 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-16 17:18:22,014 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-03-16 17:18:22,414 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-03-16 18:18:22,016 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-03-16 18:18:22,415 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-03-16 19:18:22,017 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-03-16 19:18:22,417 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-03-16 20:18:22,019 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-2) Start refreshing all users data >2014-03-16 20:18:22,418 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-03-16 21:18:22,021 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-03-16 21:18:22,419 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-03-16 22:18:22,022 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-03-16 22:18:22,420 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-03-16 23:18:22,024 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-03-16 23:18:22,421 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-03-17 00:18:22,026 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-03-17 00:18:22,422 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-03-17 01:18:22,028 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-03-17 01:18:22,424 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-03-17 02:18:22,030 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-5) Start refreshing all users data >2014-03-17 02:18:22,424 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-17 03:18:22,032 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-63) Start refreshing all users data >2014-03-17 03:18:22,425 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-03-17 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-60) Start deleteAgedOutAuditLogs >2014-03-17 03:35:35,006 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-60) Finished deleteAgedOutAuditLogs >2014-03-17 04:18:22,033 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-03-17 04:18:22,426 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-03-17 05:18:22,035 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) Start refreshing all users data >2014-03-17 05:18:22,426 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-03-17 06:18:22,037 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-63) Start refreshing all users data >2014-03-17 06:18:22,426 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-03-17 07:18:22,039 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-03-17 07:18:22,427 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-03-17 08:18:22,041 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-62) Start refreshing all users data >2014-03-17 08:18:22,428 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-03-17 09:18:22,043 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-03-17 09:18:22,429 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-03-17 10:18:22,045 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-03-17 10:18:22,431 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-03-17 11:18:22,046 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-03-17 11:18:22,431 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-17 12:18:22,047 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-03-17 12:18:22,432 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-03-17 13:18:22,049 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-03-17 13:18:22,433 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-03-17 14:18:22,050 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-03-17 14:18:22,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf. >2014-03-17 15:18:22,053 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-03-17 15:18:22,436 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-17 16:18:22,054 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-03-17 16:18:22,436 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-03-17 17:18:22,056 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-03-17 17:18:22,437 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-03-17 18:18:22,058 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-03-17 18:18:22,438 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-03-17 19:18:22,060 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-57) Start refreshing all users data >2014-03-17 19:18:22,440 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-03-17 20:18:22,062 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-76) Start refreshing all users data >2014-03-17 20:18:22,441 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-03-17 21:18:22,064 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-42) Start refreshing all users data >2014-03-17 21:18:22,442 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-03-17 22:18:22,065 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-03-17 22:18:22,444 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-03-17 23:18:22,066 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-03-17 23:18:22,445 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-03-18 00:18:22,069 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-03-18 00:18:22,446 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-03-18 01:18:22,072 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-03-18 01:18:22,447 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-03-18 02:18:22,074 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-03-18 02:18:22,448 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-03-18 03:18:22,076 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-03-18 03:18:22,449 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-03-18 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-87) Start deleteAgedOutAuditLogs >2014-03-18 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-87) Finished deleteAgedOutAuditLogs >2014-03-18 04:18:22,078 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-03-18 04:18:22,450 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-03-18 05:18:22,080 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-03-18 05:18:22,451 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-03-18 06:18:22,083 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-03-18 06:18:22,453 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-03-18 07:18:22,085 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-03-18 07:18:22,454 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-03-18 08:18:22,086 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-6) Start refreshing all users data >2014-03-18 08:18:22,456 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-03-18 09:18:22,087 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-18 09:18:22,457 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-03-18 10:18:22,088 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-03-18 10:18:22,458 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-03-18 11:18:22,090 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-03-18 11:18:22,459 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-18 12:18:22,092 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-03-18 12:18:22,460 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-03-18 13:18:22,094 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-03-18 13:18:22,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-03-18 14:18:22,096 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-18 14:18:22,463 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-03-18 15:18:22,098 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-03-18 15:18:22,464 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-03-18 16:18:22,099 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-03-18 16:18:22,465 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-03-18 17:18:22,100 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-50) Start refreshing all users data >2014-03-18 17:18:22,466 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-03-18 18:18:22,102 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-18 18:18:22,467 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-03-18 19:18:22,104 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-03-18 19:18:22,468 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-03-18 20:18:22,106 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-03-18 20:18:22,468 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-03-18 21:18:22,108 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-6) Start refreshing all users data >2014-03-18 21:18:22,470 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-03-18 22:18:22,111 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-03-18 22:18:22,470 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-03-18 23:18:22,114 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-03-18 23:18:22,472 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-03-19 00:18:22,116 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-03-19 00:18:22,475 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-03-19 01:18:22,118 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-03-19 01:18:22,476 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-03-19 02:18:22,119 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-03-19 02:18:22,476 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-19 03:18:22,121 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-03-19 03:18:22,477 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-03-19 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-80) Start deleteAgedOutAuditLogs >2014-03-19 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-80) Finished deleteAgedOutAuditLogs >2014-03-19 04:18:22,122 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-19 04:18:22,478 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-03-19 05:18:22,124 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-03-19 05:18:22,479 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-03-19 06:18:22,126 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-03-19 06:18:22,479 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-03-19 07:18:22,128 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-03-19 07:18:22,480 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-03-19 08:18:22,129 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-03-19 08:18:22,480 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-03-19 09:18:22,131 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-03-19 09:18:22,481 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-03-19 10:18:22,132 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-03-19 10:18:22,482 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-03-19 11:18:22,134 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-03-19 11:18:22,482 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-03-19 12:18:22,136 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-03-19 12:18:22,484 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-03-19 13:18:22,137 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-03-19 13:18:22,484 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-03-19 14:18:22,138 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-03-19 14:18:22,484 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-03-19 15:18:22,140 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-03-19 15:18:22,485 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-03-19 16:18:22,141 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-03-19 16:18:22,487 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-03-19 17:18:22,143 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-03-19 17:18:22,487 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-03-19 18:18:22,145 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-03-19 18:18:22,489 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-03-19 19:18:22,146 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-03-19 19:18:22,491 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-03-19 20:18:22,148 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-19 20:18:22,492 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-03-19 21:18:22,150 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-03-19 21:18:22,493 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-03-19 22:18:22,152 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-03-19 22:18:22,493 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-03-19 23:18:22,154 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-03-19 23:18:22,494 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-03-20 00:18:22,156 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-03-20 00:18:22,495 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-03-20 01:18:22,158 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-03-20 01:18:22,496 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-03-20 02:18:22,160 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-03-20 02:18:22,497 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-03-20 03:18:22,162 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-03-20 03:18:22,498 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-03-20 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-1) Start deleteAgedOutAuditLogs >2014-03-20 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-1) Finished deleteAgedOutAuditLogs >2014-03-20 04:18:22,164 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-03-20 04:18:22,498 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-03-20 05:18:22,166 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) Start refreshing all users data >2014-03-20 05:18:22,499 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-03-20 06:18:22,168 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-87) Start refreshing all users data >2014-03-20 06:18:22,502 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-03-20 07:18:22,169 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-03-20 07:18:22,502 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-03-20 08:18:22,170 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-03-20 08:18:22,503 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-03-20 09:18:22,172 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-03-20 09:18:22,504 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-03-20 10:05:21,896 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-9) Running command: LoginAdminUserCommand internal: false. >2014-03-20 10:05:21,904 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-9) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-03-20 10:05:21,969 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-1) Running command: LoginUserCommand internal: false. >2014-03-20 10:05:22,021 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-03-20 10:18:22,175 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-54) Start refreshing all users data >2014-03-20 10:18:22,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-03-20 11:18:22,175 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-03-20 11:18:22,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-03-20 12:18:22,176 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-03-20 12:18:22,507 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-03-20 13:18:22,178 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-03-20 13:18:22,508 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-03-20 14:18:22,179 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-03-20 14:18:22,509 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-03-20 15:18:22,181 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-03-20 15:18:22,510 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-03-20 16:18:22,182 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-81) Start refreshing all users data >2014-03-20 16:18:22,512 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-03-20 17:18:22,184 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-03-20 17:18:22,514 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-03-20 18:18:22,185 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) Start refreshing all users data >2014-03-20 18:18:22,514 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-03-20 19:18:22,187 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-18) Start refreshing all users data >2014-03-20 19:18:22,515 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-03-20 20:18:22,190 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-03-20 20:18:22,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-03-20 21:18:22,193 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-03-20 21:18:22,518 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-03-20 22:18:22,195 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-03-20 22:18:22,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-03-20 23:18:22,197 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) Start refreshing all users data >2014-03-20 23:18:22,522 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-03-21 00:18:22,200 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-87) Start refreshing all users data >2014-03-21 00:18:22,522 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-03-21 01:18:22,201 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-03-21 01:18:22,523 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-03-21 02:18:22,203 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-03-21 02:18:22,524 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-03-21 03:18:22,205 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-03-21 03:18:22,525 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-03-21 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-87) Start deleteAgedOutAuditLogs >2014-03-21 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-87) Finished deleteAgedOutAuditLogs >2014-03-21 04:18:22,207 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-03-21 04:18:22,526 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-03-21 05:18:22,209 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-03-21 05:18:22,527 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-03-21 06:18:22,211 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-66) Start refreshing all users data >2014-03-21 06:18:22,528 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-03-21 07:18:22,213 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-03-21 07:18:22,529 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-03-21 08:18:22,215 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-03-21 08:18:22,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-03-21 09:18:22,217 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-03-21 09:18:22,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-03-21 10:18:22,219 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-03-21 10:18:22,532 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-03-21 11:18:22,220 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-03-21 11:18:22,532 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-03-21 12:18:22,221 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-03-21 12:18:22,533 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-03-21 13:18:22,223 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-03-21 13:18:22,534 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-03-21 14:18:22,225 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-03-21 14:18:22,534 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-03-21 15:18:22,226 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-03-21 15:18:22,536 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-03-21 16:18:22,229 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-03-21 16:18:22,537 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-21 17:18:22,230 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-03-21 17:18:22,541 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-03-21 18:18:22,232 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-03-21 18:18:22,542 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-03-21 19:18:22,234 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-03-21 19:18:22,544 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-21 20:18:22,236 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-03-21 20:18:22,545 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-03-21 21:18:22,237 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-03-21 21:18:22,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-03-21 22:18:22,238 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-03-21 22:18:22,548 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-03-21 23:18:22,240 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-03-21 23:18:22,549 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-22 00:18:22,242 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-03-22 00:18:22,551 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-03-22 01:18:22,243 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-03-22 01:18:22,552 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-03-22 02:18:22,245 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-03-22 02:18:22,554 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-22 03:18:22,246 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-03-22 03:18:22,554 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-03-22 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-89) Start deleteAgedOutAuditLogs >2014-03-22 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-89) Finished deleteAgedOutAuditLogs >2014-03-22 04:18:22,248 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-03-22 04:18:22,555 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-03-22 05:18:22,251 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-03-22 05:18:22,556 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-03-22 06:18:22,252 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-50) Start refreshing all users data >2014-03-22 06:18:22,556 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-03-22 07:18:22,254 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-03-22 07:18:22,557 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-03-22 08:18:22,255 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-03-22 08:18:22,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-03-22 09:18:22,256 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-03-22 09:18:22,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-03-22 10:18:22,258 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-03-22 10:18:22,559 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-03-22 11:18:22,260 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-03-22 11:18:22,560 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-03-22 12:18:22,262 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-03-22 12:18:22,561 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-03-22 13:18:22,264 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-22 13:18:22,562 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-03-22 14:18:22,265 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-03-22 14:18:22,564 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-03-22 15:18:22,266 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-54) Start refreshing all users data >2014-03-22 15:18:22,564 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-03-22 16:18:22,267 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-22 16:18:22,566 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-03-22 17:18:22,268 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-63) Start refreshing all users data >2014-03-22 17:18:22,566 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-03-22 18:18:22,270 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-6) Start refreshing all users data >2014-03-22 18:18:22,567 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-03-22 19:18:22,272 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-03-22 19:18:22,569 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-03-22 20:18:22,274 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-03-22 20:18:22,570 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-03-22 21:18:22,276 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-03-22 21:18:22,571 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-03-22 22:18:22,278 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-03-22 22:18:22,572 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-03-22 23:18:22,279 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-03-22 23:18:22,573 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-03-23 00:18:22,280 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-03-23 00:18:22,575 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-03-23 01:18:22,281 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-03-23 01:18:22,576 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-03-23 02:18:22,282 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-03-23 02:18:22,578 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-03-23 03:18:22,283 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-03-23 03:18:22,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-23 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-53) Start deleteAgedOutAuditLogs >2014-03-23 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-53) Finished deleteAgedOutAuditLogs >2014-03-23 04:18:22,284 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-03-23 04:18:22,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-03-23 05:18:22,286 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-03-23 05:18:22,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-03-23 06:18:22,288 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-67) Start refreshing all users data >2014-03-23 06:18:22,581 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-03-23 07:18:22,289 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) Start refreshing all users data >2014-03-23 07:18:22,583 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-03-23 08:18:22,291 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-03-23 08:18:22,584 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-03-23 09:18:22,292 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-03-23 09:18:22,585 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-03-23 10:18:22,293 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-03-23 10:18:22,587 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-03-23 11:18:22,294 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-87) Start refreshing all users data >2014-03-23 11:18:22,588 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-03-23 12:18:22,296 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-03-23 12:18:22,589 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-03-23 13:18:22,299 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-03-23 13:18:22,590 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-03-23 14:18:22,302 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-89) Start refreshing all users data >2014-03-23 14:18:22,592 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-03-23 15:18:22,305 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-03-23 15:18:22,592 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-03-23 16:18:22,306 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-03-23 16:18:22,593 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-03-23 17:18:22,308 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-03-23 17:18:22,594 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-03-23 18:18:22,310 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-03-23 18:18:22,595 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-23 19:18:22,312 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-03-23 19:18:22,596 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-03-23 20:18:22,314 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-03-23 20:18:22,598 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-03-23 21:18:22,315 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-03-23 21:18:22,598 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-03-23 22:18:22,317 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-03-23 22:18:22,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-03-23 23:18:22,318 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-03-23 23:18:22,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-03-24 00:18:22,321 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-03-24 00:18:22,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-03-24 01:18:22,323 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-03-24 01:18:22,600 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-03-24 02:18:22,324 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-03-24 02:18:22,601 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-03-24 03:18:22,326 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-03-24 03:18:22,601 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-03-24 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-92) Start deleteAgedOutAuditLogs >2014-03-24 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-92) Finished deleteAgedOutAuditLogs >2014-03-24 04:18:22,327 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-03-24 04:18:22,603 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-03-24 05:18:22,329 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-62) Start refreshing all users data >2014-03-24 05:18:22,604 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-03-24 06:18:22,331 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-03-24 06:18:22,605 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-03-24 07:18:22,333 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-24 07:18:22,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-03-24 08:18:22,335 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-03-24 08:18:22,607 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-03-24 09:18:22,337 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-03-24 09:18:22,608 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-03-24 10:18:22,339 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-03-24 10:18:22,609 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-03-24 11:18:22,341 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-66) Start refreshing all users data >2014-03-24 11:18:22,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-03-24 12:18:22,344 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-03-24 12:18:22,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-03-24 13:18:22,346 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-03-24 13:18:22,612 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-03-24 14:18:22,348 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-03-24 14:18:22,613 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-03-24 15:18:22,350 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-03-24 15:18:22,614 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-03-24 16:18:22,351 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-03-24 16:18:22,616 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-03-24 17:18:22,353 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-03-24 17:18:22,616 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-03-24 18:18:22,355 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-63) Start refreshing all users data >2014-03-24 18:18:22,618 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-03-24 19:18:22,356 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-03-24 19:18:22,618 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-03-24 20:18:22,359 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-03-24 20:18:22,620 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-03-24 21:18:22,360 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-03-24 21:18:22,621 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-24 22:18:22,362 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-03-24 22:18:22,621 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VMs/Templates Ovf. >2014-03-24 23:18:22,364 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-24 23:18:22,622 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-03-25 00:18:22,366 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-03-25 00:18:22,622 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-03-25 01:18:22,373 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-57) Start refreshing all users data >2014-03-25 01:18:22,623 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-03-25 02:18:22,375 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-25) Start refreshing all users data >2014-03-25 02:18:22,624 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-03-25 03:18:22,377 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-03-25 03:18:22,625 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-03-25 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-91) Start deleteAgedOutAuditLogs >2014-03-25 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-91) Finished deleteAgedOutAuditLogs >2014-03-25 04:18:22,378 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-03-25 04:18:22,625 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-03-25 05:18:22,380 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-03-25 05:18:22,626 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-03-25 06:18:22,382 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-81) Start refreshing all users data >2014-03-25 06:18:22,628 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-25 07:18:22,383 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-03-25 07:18:22,628 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-03-25 08:18:22,384 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-03-25 08:18:22,629 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-03-25 09:18:22,387 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-03-25 09:18:22,630 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-25 10:18:22,389 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-03-25 10:18:22,631 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-03-25 11:18:22,391 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-03-25 11:18:22,632 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-03-25 12:18:22,392 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-25 12:18:22,634 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-03-25 13:18:22,394 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-25 13:18:22,635 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-03-25 14:18:22,396 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-03-25 14:18:22,635 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VMs/Templates Ovf. >2014-03-25 15:18:22,398 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-25 15:18:22,636 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-03-25 16:18:22,400 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-03-25 16:18:22,637 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-03-25 17:18:22,402 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-03-25 17:18:22,638 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-25 18:18:22,404 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-89) Start refreshing all users data >2014-03-25 18:18:22,639 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-25 19:18:22,406 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-03-25 19:18:22,640 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-03-25 20:18:22,407 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-03-25 20:18:22,641 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-03-25 21:18:22,409 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-20) Start refreshing all users data >2014-03-25 21:18:22,642 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-03-25 22:18:22,411 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-03-25 22:18:22,644 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-03-25 23:18:22,413 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-03-25 23:18:22,645 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-03-26 00:18:22,415 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-03-26 00:18:22,645 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-03-26 01:18:22,417 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-03-26 01:18:22,646 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-03-26 02:18:22,419 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-03-26 02:18:22,647 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-03-26 03:18:22,420 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-03-26 03:18:22,648 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-03-26 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-36) Start deleteAgedOutAuditLogs >2014-03-26 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-36) Finished deleteAgedOutAuditLogs >2014-03-26 04:18:22,421 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-03-26 04:18:22,649 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-03-26 05:18:22,423 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-03-26 05:18:22,650 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-03-26 06:18:22,425 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-03-26 06:18:22,651 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-03-26 07:18:22,427 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-03-26 07:18:22,652 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-03-26 08:18:22,428 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-03-26 08:18:22,653 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-03-26 09:18:22,429 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-88) Start refreshing all users data >2014-03-26 09:18:22,654 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-03-26 10:18:22,430 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-03-26 10:18:22,654 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-03-26 11:18:22,432 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-03-26 11:18:22,655 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-03-26 12:18:22,433 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-26 12:18:22,655 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-03-26 13:18:22,435 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-03-26 13:18:22,656 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-03-26 14:18:22,437 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-03-26 14:18:22,657 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-03-26 15:18:22,439 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-03-26 15:18:22,658 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-26 16:18:22,440 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-03-26 16:18:22,660 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-03-26 17:18:22,443 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-03-26 17:18:22,661 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-03-26 18:18:22,445 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-03-26 18:18:22,662 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-03-26 19:18:22,448 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-66) Start refreshing all users data >2014-03-26 19:18:22,663 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-03-26 20:18:22,450 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-03-26 20:18:22,664 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-03-26 21:18:22,452 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-6) Start refreshing all users data >2014-03-26 21:18:22,665 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-03-26 22:18:22,454 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-03-26 22:18:22,666 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-03-26 23:18:22,455 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-03-26 23:18:22,667 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-03-27 00:18:22,456 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-03-27 00:18:22,669 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-03-27 01:18:22,457 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-03-27 01:18:22,670 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-27 02:18:22,458 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-03-27 02:18:22,671 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-03-27 03:18:22,461 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-03-27 03:18:22,672 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-03-27 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-47) Start deleteAgedOutAuditLogs >2014-03-27 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-47) Finished deleteAgedOutAuditLogs >2014-03-27 04:18:22,462 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-03-27 04:18:22,672 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-03-27 05:18:22,464 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-03-27 05:18:22,673 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-03-27 06:18:22,466 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-03-27 06:18:22,674 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-03-27 07:18:22,468 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-03-27 07:18:22,675 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-03-27 08:18:22,470 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-03-27 08:18:22,676 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-03-27 09:18:22,474 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) Start refreshing all users data >2014-03-27 09:18:22,677 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-03-27 10:18:22,476 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-27 10:18:22,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-03-27 11:18:22,476 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-03-27 11:18:22,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-03-27 12:18:22,478 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-03-27 12:18:22,679 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-03-27 12:45:06,241 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginAdminUserCommand internal: false. >2014-03-27 12:45:06,254 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-03-27 12:45:07,883 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-03-27 12:45:07,889 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-03-27 12:46:48,231 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-10) [4e21d038] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-03-27 12:46:48,291 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-8) [4e21d038] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:46:48,291 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-8) [4e21d038] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center Lok >2014-03-27 12:46:48,292 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-8) [4e21d038] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center Lok >2014-03-27 12:46:48,310 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-8) [4e21d038] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5705a1d6 >2014-03-27 12:46:48,323 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-8) [4e21d038] FINISH, SetVdsStatusVDSCommand, log id: 5705a1d6 >2014-03-27 12:46:48,324 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-8) [4e21d038] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: e7c85d2 >2014-03-27 12:46:48,445 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (pool-4-thread-8) [4e21d038] Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-03-27 12:46:48,465 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-8) [15c5d877] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:46:48,472 WARN [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-8) [7e5909d9] CanDoAction of action HandleVdsVersion failed. Reasons:VDS_CANNOT_CHECK_VERSION_HOST_NON_RESPONSIVE >2014-03-27 12:46:48,472 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (pool-4-thread-8) [7e5909d9] Failed to activate VDS = 53fd133a-9dc6-11e2-abd9-001a4a223991 with error: java.net.ConnectException: Connection refused. >2014-03-27 12:46:48,473 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-8) [7e5909d9] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: e7c85d2 >2014-03-27 12:46:48,512 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-8) [7e5909d9] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center Lok >2014-03-27 12:46:48,542 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-8) [7e5909d9] Correlation ID: 4e21d038, Job ID: 74e6f032-4071-49db-9bd3-803284ad061c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was activated by admin@internal. >2014-03-27 12:46:48,567 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-8) [7e5909d9] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-03-27 12:46:48,596 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-54) Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-03-27 12:46:48,626 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-54) Failed to refresh VDS , vds = 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-03-27 12:46:51,651 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-12) Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-03-27 12:46:51,652 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-12) Server failed to respond, vds_id = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds_name = 10.34.27.209, vm_count = 0, spm_status = None, non-responsive_timeout (seconds) = 60, error = java.net.ConnectException: Connection refused >2014-03-27 12:46:51,661 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.ConnectException: Connection refused > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:102) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:532) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:485) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:336) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:235) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy59.getVdsCapabilities(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:225) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Host 10.34.27.209 is non responsive. >2014-03-27 12:46:51,673 INFO [org.ovirt.engine.core.bll.VdsEventListener] (pool-4-thread-12) ResourceManager::vdsNotResponding entered for Host 53fd133a-9dc6-11e2-abd9-001a4a223991, 10.34.27.209 >2014-03-27 12:46:51,707 INFO [org.ovirt.engine.core.bll.SshSoftFencingCommand] (pool-4-thread-12) [5ae675dc] Running command: SshSoftFencingCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:46:52,394 ERROR [org.ovirt.engine.core.bll.SshSoftFencingCommand] (pool-4-thread-12) [5ae675dc] SSH connection to host 10.34.27.209 failed: javax.naming.AuthenticationException: SSH authentication to 'root@10.34.27.209' failed. Please verify provided credentials. Make sure key is authorized at host: javax.naming.AuthenticationException: SSH authentication to 'root@10.34.27.209' failed. Please verify provided credentials. Make sure key is authorized at host > at org.ovirt.engine.core.utils.ssh.SSHClient.authenticate(SSHClient.java:359) [utils.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeSshSoftFencingCommand(SshSoftFencingCommand.java:74) [bll.jar:] > at org.ovirt.engine.core.bll.SshSoftFencingCommand.executeCommand(SshSoftFencingCommand.java:43) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:636) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view10.runInternalAction(Unknown Source) [bll.jar:] > at org.ovirt.engine.core.bll.VdsEventListener$1.run(VdsEventListener.java:141) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-03-27 12:46:52,442 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-12) [2c3734d4] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS_FENCE >, sharedLocks= ] >2014-03-27 12:46:52,484 ERROR [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-12) [2c3734d4] Failed to run Fence script on vds:10.34.27.209, VMs moved to UnKnown instead. >2014-03-27 12:46:52,493 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-12) [2c3734d4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 became non responsive. It has no power management configured. Please check the host status, manually reboot it, and click "Confirm Host Has Been Rebooted" >2014-03-27 12:46:52,493 WARN [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-12) [2c3734d4] CanDoAction of action VdsNotRespondingTreatment failed. Reasons:VAR__ACTION__RESTART,VDS_FENCE_DISABLED >2014-03-27 12:46:52,493 INFO [org.ovirt.engine.core.bll.VdsNotRespondingTreatmentCommand] (pool-4-thread-12) [2c3734d4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS_FENCE >, sharedLocks= ] >2014-03-27 12:46:54,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-03-27 12:46:57,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-6) Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-03-27 12:47:00,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-03-27 12:47:01,763 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (pool-4-thread-16) [758b577e] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:47:01,829 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (pool-4-thread-16) [758b577e] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:47:01,831 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-16) [758b577e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Maintenance, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 50b2ee77 >2014-03-27 12:47:01,839 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-16) [758b577e] FINISH, SetVdsStatusVDSCommand, log id: 50b2ee77 >2014-03-27 12:47:01,849 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-16) [758b577e] Correlation ID: 758b577e, Job ID: 2bf476de-2f6a-408f-adb2-362f25c58cf8, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was switched to Maintenance mode by admin@internal. >2014-03-27 12:48:17,161 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-14) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:17,161 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-14) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:17,162 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-14) [3ca144ef] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:48:17,274 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-14) [3ca144ef] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 484bf465 >2014-03-27 12:48:17,285 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-14) [3ca144ef] FINISH, SetVdsStatusVDSCommand, log id: 484bf465 >2014-03-27 12:48:17,286 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-14) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:17,286 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-14) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:17,293 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-14) [3ca144ef] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-03-27 12:48:17,298 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-14) [3ca144ef] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-03-27 12:48:17,303 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-17) [3ca144ef] Running command: InstallVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:48:17,304 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-17) [3ca144ef] Before Installation host 53fd133a-9dc6-11e2-abd9-001a4a223991, 10.34.27.209 >2014-03-27 12:48:17,342 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-17) [3ca144ef] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4ecef824 >2014-03-27 12:48:17,351 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-14) [3ca144ef] Correlation ID: 3ca144ef, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 configuration was updated by admin@internal. >2014-03-27 12:48:17,354 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-17) [3ca144ef] FINISH, SetVdsStatusVDSCommand, log id: 4ecef824 >2014-03-27 12:48:17,431 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-17) [3ca144ef] Installation 10.34.27.209: Connected to host 10.34.27.209 with SSH key fingerprint: a2:dd:38:20:0e:c2:a2:7d:36:68:f0:b5:4d:35:2e:ef >2014-03-27 12:48:17,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-17) [3ca144ef] Correlation ID: 3ca144ef, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Connected to host 10.34.27.209 with SSH key fingerprint: a2:dd:38:20:0e:c2:a2:7d:36:68:f0:b5:4d:35:2e:ef. >2014-03-27 12:48:17,570 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (pool-4-thread-17) [3ca144ef] Error during host 10.34.27.209 install: javax.naming.AuthenticationException: SSH authentication to 'root@10.34.27.209' failed. Please verify provided credentials. Make sure key is authorized at host > at org.ovirt.engine.core.utils.ssh.SSHClient.authenticate(SSHClient.java:359) [utils.jar:] > at org.ovirt.engine.core.utils.ssh.SSHDialog.authenticate(SSHDialog.java:264) [utils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1024) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.installHost(InstallVdsCommand.java:192) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:105) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:175) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:156) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:94) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-03-27 12:48:17,570 ERROR [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-17) [3ca144ef] Host installation failed for host 53fd133a-9dc6-11e2-abd9-001a4a223991, 10.34.27.209.: javax.naming.AuthenticationException: SSH authentication to 'root@10.34.27.209' failed. Please verify provided credentials. Make sure key is authorized at host > at org.ovirt.engine.core.utils.ssh.SSHClient.authenticate(SSHClient.java:359) [utils.jar:] > at org.ovirt.engine.core.utils.ssh.SSHDialog.authenticate(SSHDialog.java:264) [utils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1024) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.installHost(InstallVdsCommand.java:192) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:105) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:175) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:156) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:94) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-03-27 12:48:17,577 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-17) [3ca144ef] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=InstallFailed, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 37c330e9 >2014-03-27 12:48:17,583 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-17) [3ca144ef] FINISH, SetVdsStatusVDSCommand, log id: 37c330e9 >2014-03-27 12:48:17,597 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-17) [3ca144ef] Correlation ID: 3ca144ef, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 installation failed. SSH authentication to 'root@10.34.27.209' failed. Please verify provided credentials. Make sure key is authorized at host. >2014-03-27 12:48:17,597 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-17) [3ca144ef] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-03-27 12:48:39,382 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:39,385 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:39,385 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-1) [5884be21] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:48:39,393 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-1) [5884be21] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 69fe7dae >2014-03-27 12:48:39,400 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-1) [5884be21] FINISH, SetVdsStatusVDSCommand, log id: 69fe7dae >2014-03-27 12:48:39,401 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:39,401 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-1) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:39,405 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-1) [5884be21] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-03-27 12:48:39,413 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [5884be21] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-03-27 12:48:39,415 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-18) [5884be21] Running command: InstallVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:48:39,415 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-18) [5884be21] Before Installation host 53fd133a-9dc6-11e2-abd9-001a4a223991, 10.34.27.209 >2014-03-27 12:48:39,419 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-18) [5884be21] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 8db477d >2014-03-27 12:48:39,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [5884be21] Correlation ID: 5884be21, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 configuration was updated by admin@internal. >2014-03-27 12:48:39,428 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-18) [5884be21] FINISH, SetVdsStatusVDSCommand, log id: 8db477d >2014-03-27 12:48:39,504 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-18) [5884be21] Installation 10.34.27.209: Connected to host 10.34.27.209 with SSH key fingerprint: a2:dd:38:20:0e:c2:a2:7d:36:68:f0:b5:4d:35:2e:ef >2014-03-27 12:48:39,511 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-18) [5884be21] Correlation ID: 5884be21, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Connected to host 10.34.27.209 with SSH key fingerprint: a2:dd:38:20:0e:c2:a2:7d:36:68:f0:b5:4d:35:2e:ef. >2014-03-27 12:48:41,914 ERROR [org.ovirt.engine.core.bll.VdsDeploy] (pool-4-thread-18) [5884be21] Error during host 10.34.27.209 install: javax.naming.AuthenticationException: SSH authentication to 'root@10.34.27.209' failed. Please verify provided credentials. Make sure host is configured for password authentication > at org.ovirt.engine.core.utils.ssh.SSHClient.authenticate(SSHClient.java:359) [utils.jar:] > at org.ovirt.engine.core.utils.ssh.SSHDialog.authenticate(SSHDialog.java:264) [utils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1024) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.installHost(InstallVdsCommand.java:192) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:105) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:175) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:156) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:94) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-03-27 12:48:41,915 ERROR [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-18) [5884be21] Host installation failed for host 53fd133a-9dc6-11e2-abd9-001a4a223991, 10.34.27.209.: javax.naming.AuthenticationException: SSH authentication to 'root@10.34.27.209' failed. Please verify provided credentials. Make sure host is configured for password authentication > at org.ovirt.engine.core.utils.ssh.SSHClient.authenticate(SSHClient.java:359) [utils.jar:] > at org.ovirt.engine.core.utils.ssh.SSHDialog.authenticate(SSHDialog.java:264) [utils.jar:] > at org.ovirt.engine.core.bll.VdsDeploy.execute(VdsDeploy.java:1024) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.installHost(InstallVdsCommand.java:192) [bll.jar:] > at org.ovirt.engine.core.bll.InstallVdsCommand.executeCommand(InstallVdsCommand.java:105) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) [bll.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) [utils.jar:] > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) [utils.jar:] > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.executeValidatedCommand(MultipleActionsRunner.java:175) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.RunCommands(MultipleActionsRunner.java:156) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner$1.run(MultipleActionsRunner.java:94) [bll.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalWrapperRunnable.run(ThreadPoolUtil.java:71) [utils.jar:] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-03-27 12:48:41,919 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-18) [5884be21] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=InstallFailed, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7871ec07 >2014-03-27 12:48:41,925 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-18) [5884be21] FINISH, SetVdsStatusVDSCommand, log id: 7871ec07 >2014-03-27 12:48:41,931 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-18) [5884be21] Correlation ID: 5884be21, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 installation failed. SSH authentication to 'root@10.34.27.209' failed. Please verify provided credentials. Make sure host is configured for password authentication. >2014-03-27 12:48:41,931 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-18) [5884be21] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-03-27 12:48:59,631 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:59,632 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:59,632 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-10) [62ae86f4] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:48:59,638 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-10) [62ae86f4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3b91fd8b >2014-03-27 12:48:59,641 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (ajp-/127.0.0.1:8702-10) [62ae86f4] FINISH, SetVdsStatusVDSCommand, log id: 3b91fd8b >2014-03-27 12:48:59,641 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:59,641 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-10) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-03-27 12:48:59,642 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (ajp-/127.0.0.1:8702-10) [62ae86f4] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-03-27 12:48:59,649 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [62ae86f4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-03-27 12:48:59,650 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-19) [62ae86f4] Running command: InstallVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:48:59,651 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-19) [62ae86f4] Before Installation host 53fd133a-9dc6-11e2-abd9-001a4a223991, 10.34.27.209 >2014-03-27 12:48:59,655 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) [62ae86f4] Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 configuration was updated by admin@internal. >2014-03-27 12:48:59,656 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-19) [62ae86f4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5d5a45bc >2014-03-27 12:48:59,661 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-19) [62ae86f4] FINISH, SetVdsStatusVDSCommand, log id: 5d5a45bc >2014-03-27 12:48:59,729 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-19) [62ae86f4] Installation 10.34.27.209: Connected to host 10.34.27.209 with SSH key fingerprint: a2:dd:38:20:0e:c2:a2:7d:36:68:f0:b5:4d:35:2e:ef >2014-03-27 12:48:59,735 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-19) [62ae86f4] Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Connected to host 10.34.27.209 with SSH key fingerprint: a2:dd:38:20:0e:c2:a2:7d:36:68:f0:b5:4d:35:2e:ef. >2014-03-27 12:48:59,964 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-4-thread-19) [62ae86f4] Installation of 10.34.27.209. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-03-27 12:48:59,976 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-4-thread-19) SSH execute root@10.34.27.209 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-03-27 12:49:02,006 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Initializing >2014-03-27 12:49:02,013 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Initializing. >2014-03-27 12:49:02,028 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Environment setup >2014-03-27 12:49:02,033 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Environment setup. >2014-03-27 12:49:02,046 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Environment packages setup >2014-03-27 12:49:02,049 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Environment packages setup. >2014-03-27 12:49:06,817 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Programs detection >2014-03-27 12:49:06,821 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Programs detection. >2014-03-27 12:49:06,822 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Environment customization >2014-03-27 12:49:06,825 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Environment customization. >2014-03-27 12:49:06,879 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Logs at host located at: '/tmp/ovirt-host-deploy-20140327124426.log' >2014-03-27 12:49:06,885 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Logs at host located at: '/tmp/ovirt-host-deploy-20140327124426.log'. >2014-03-27 12:49:06,973 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.34.27.209 reports unique id A9ED61F2-B5C6-DC11-BBDA-C438AF16001C >2014-03-27 12:49:06,999 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id A9ED61F2-B5C6-DC11-BBDA-C438AF16001C to Host 10.34.27.209 >2014-03-27 12:49:07,045 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Enforcing host reboot >2014-03-27 12:49:07,049 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Enforcing host reboot. >2014-03-27 12:49:07,079 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Setup validation >2014-03-27 12:49:07,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Setup validation. >2014-03-27 12:49:07,137 WARN [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Cannot validate host name settings, reason: resolved host does not match any of the local addresses >2014-03-27 12:49:07,141 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 installation in progress . Cannot validate host name settings, reason: resolved host does not match any of the local addresses. >2014-03-27 12:49:07,154 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Hardware supports virtualization >2014-03-27 12:49:07,159 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Hardware supports virtualization. >2014-03-27 12:49:07,667 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Transaction setup >2014-03-27 12:49:07,671 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Transaction setup. >2014-03-27 12:49:07,747 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Misc configuration >2014-03-27 12:49:07,752 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Misc configuration. >2014-03-27 12:49:07,753 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Package installation >2014-03-27 12:49:07,757 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Package installation. >2014-03-27 12:49:11,107 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Yum Status: Downloading Packages >2014-03-27 12:49:11,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Yum Status: Downloading Packages. >2014-03-27 12:49:11,186 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Yum Download/Verify: tuned-0.2.19-13.el6.noarch >2014-03-27 12:49:11,191 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Yum Download/Verify: tuned-0.2.19-13.el6.noarch. >2014-03-27 12:49:11,192 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Yum Status: Check Package Signatures >2014-03-27 12:49:11,198 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Yum Status: Check Package Signatures. >2014-03-27 12:49:11,198 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Yum Status: Running Test Transaction >2014-03-27 12:49:11,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Yum Status: Running Test Transaction. >2014-03-27 12:49:11,508 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Yum Status: Running Transaction >2014-03-27 12:49:11,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Yum Status: Running Transaction. >2014-03-27 12:49:12,263 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Yum install: 1/1: tuned-0.2.19-13.el6.noarch >2014-03-27 12:49:12,267 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Yum install: 1/1: tuned-0.2.19-13.el6.noarch. >2014-03-27 12:49:21,067 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Yum Verify: 1/1: tuned.noarch 0:0.2.19-13.el6 - u >2014-03-27 12:49:21,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Yum Verify: 1/1: tuned.noarch 0:0.2.19-13.el6 - u. >2014-03-27 12:49:21,520 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Misc configuration >2014-03-27 12:49:21,527 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Misc configuration. >2014-03-27 12:49:23,482 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Enrolling certificate >2014-03-27 12:49:23,487 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Enrolling certificate. >2014-03-27 12:49:24,845 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Transaction commit >2014-03-27 12:49:24,850 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Transaction commit. >2014-03-27 12:49:25,005 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Closing up >2014-03-27 12:49:25,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Closing up. >2014-03-27 12:49:26,510 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Reboot scheduled >2014-03-27 12:49:26,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Reboot scheduled. >2014-03-27 12:49:26,514 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Pre-termination >2014-03-27 12:49:26,518 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Pre-termination. >2014-03-27 12:49:26,534 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Reboot scheduled >2014-03-27 12:49:26,538 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Reboot scheduled. >2014-03-27 12:49:26,538 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140327124926-10.34.27.209-62ae86f4.log' >2014-03-27 12:49:26,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140327124926-10.34.27.209-62ae86f4.log'. >2014-03-27 12:49:26,845 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.27.209: Stage: Termination >2014-03-27 12:49:26,850 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.27.209. Stage: Termination. >2014-03-27 12:49:27,050 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-19) [62ae86f4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Reboot, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4434ce1e >2014-03-27 12:49:27,056 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-19) [62ae86f4] FINISH, SetVdsStatusVDSCommand, log id: 4434ce1e >2014-03-27 12:49:27,058 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-19) [62ae86f4] After Installation host 10.34.27.209, VDS >2014-03-27 12:49:27,063 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-19) [62ae86f4] Correlation ID: 62ae86f4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 installed >2014-03-27 12:49:27,063 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-19) [62ae86f4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-03-27 12:49:27,063 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-20) [62ae86f4] Waiting 300 seconds, for server to finish reboot process. >2014-03-27 12:54:27,066 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-20) [62ae86f4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonResponsive, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7d27381f >2014-03-27 12:54:27,069 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-20) [62ae86f4] FINISH, SetVdsStatusVDSCommand, log id: 7d27381f >2014-03-27 12:54:29,237 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1fbb181e >2014-03-27 12:54:29,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GetHardwareInfoVDSCommand, log id: 1fbb181e >2014-03-27 12:54:29,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 73c893c8 >2014-03-27 12:54:30,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GetHardwareInfoVDSCommand, log id: 73c893c8 >2014-03-27 12:54:30,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-03-27 12:54:30,444 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-44) [1a7421bd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-03-27 12:54:30,448 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-03-27 12:54:30,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, storageType = LOCALFS, connectionList = [{ id: 48bb8365-f600-4eba-8ed7-e823810d8790, connection: /mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 15ea45ed >2014-03-27 12:54:30,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] FINISH, ConnectStorageServerVDSCommand, return: {48bb8365-f600-4eba-8ed7-e823810d8790=0}, log id: 15ea45ed >2014-03-27 12:54:30,848 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] Host 10.34.27.209 storage connection was succeeded >2014-03-27 12:54:30,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5b3714a7 >2014-03-27 12:54:31,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 5b3714a7 >2014-03-27 12:54:31,313 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] Host 10.34.27.209 storage connection was succeeded >2014-03-27 12:54:31,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 25298bf1 >2014-03-27 12:54:31,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] FINISH, ConnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 25298bf1 >2014-03-27 12:54:31,617 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] Host 10.34.27.209 storage connection was succeeded >2014-03-27 12:54:31,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-30) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, vds_spm_id = 1, masterDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterVersion = 1), log id: 620c8e94 >2014-03-27 12:54:40,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-30) FINISH, ConnectStoragePoolVDSCommand, log id: 620c8e94 >2014-03-27 12:54:40,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [2d1ac73e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-03-27 12:54:40,378 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [5df5070e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:54:40,393 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [6523099e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-03-27 12:54:45,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool Lok >2014-03-27 12:54:45,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) SpmStatus on vds 53fd133a-9dc6-11e2-abd9-001a4a223991: Free >2014-03-27 12:54:45,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) starting spm on vds 10.34.27.209, storage pool Lok, prevId 1, LVER 14 >2014-03-27 12:54:45,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, prevId=1, prevLVER=14, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b1fabb7 >2014-03-27 12:54:46,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling started: taskId = 695e32a7-05aa-4220-ad61-0fc49aa031cf >2014-03-27 12:54:47,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling ended: taskId = 695e32a7-05aa-4220-ad61-0fc49aa031cf task status = finished >2014-03-27 12:54:47,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling ended, spm status: SPM >2014-03-27 12:54:47,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-64) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=695e32a7-05aa-4220-ad61-0fc49aa031cf), log id: 218cd0fc >2014-03-27 12:54:47,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, HSMClearTaskVDSCommand, log id: 218cd0fc >2014-03-27 12:54:47,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@463789d3, log id: 6b1fabb7 >2014-03-27 12:54:47,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) Initialize Irs proxy from vds: 10.34.27.209 >2014-03-27 12:54:47,938 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host 10.34.27.209 (Address: 10.34.27.209). >2014-03-27 12:54:48,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-47) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4fab084b >2014-03-27 12:54:48,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-47) -- executeIrsBrokerCommand: Attempting on storage pool 19dcb0b3-4c52-4803-a39e-992b1392da62 >2014-03-27 12:54:48,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (pool-4-thread-47) START, HSMGetAllTasksInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 10992510 >2014-03-27 12:54:48,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (pool-4-thread-47) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 10992510 >2014-03-27 12:54:48,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-47) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 4fab084b >2014-03-27 12:54:48,533 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-47) Discovered no tasks on Storage Pool Lok >2014-03-27 13:18:22,480 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-27 13:18:22,680 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-03-27 13:18:22,681 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-03-27 13:18:22,691 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-03-27 13:18:22,692 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-03-27 13:18:22,697 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-03-27 13:18:22,698 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 13:18:22,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 13:23:21,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 714345f1 >2014-03-27 13:23:21,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3c604302 >2014-03-27 13:23:22,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c604302 >2014-03-27 13:23:22,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 714345f1 >2014-03-27 13:23:22,095 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-27 14:18:22,482 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-03-27 14:18:22,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-03-27 14:18:22,702 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VM OVFs in Data Center Lok >2014-03-27 14:18:22,707 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully updated VM OVFs in Data Center Lok >2014-03-27 14:18:22,707 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update template OVFs in Data Center Lok >2014-03-27 14:18:22,711 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully updated templates OVFs in Data Center Lok >2014-03-27 14:18:22,711 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 14:18:22,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 14:23:22,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2256c905 >2014-03-27 14:23:22,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5bf0b1bd >2014-03-27 14:23:22,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5bf0b1bd >2014-03-27 14:23:22,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2256c905 >2014-03-27 14:23:22,339 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-27 15:18:22,484 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-03-27 15:18:22,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-03-27 15:18:22,714 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VM OVFs in Data Center Lok >2014-03-27 15:18:22,722 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated VM OVFs in Data Center Lok >2014-03-27 15:18:22,722 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update template OVFs in Data Center Lok >2014-03-27 15:18:22,723 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated templates OVFs in Data Center Lok >2014-03-27 15:18:22,723 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 15:18:22,727 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 15:23:22,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2b77cfaf >2014-03-27 15:23:22,352 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 46e40074 >2014-03-27 15:23:22,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 46e40074 >2014-03-27 15:23:22,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2b77cfaf >2014-03-27 15:23:22,597 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-27 16:18:22,486 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-03-27 16:18:22,728 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-03-27 16:18:22,728 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VM OVFs in Data Center Lok >2014-03-27 16:18:22,735 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated VM OVFs in Data Center Lok >2014-03-27 16:18:22,735 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update template OVFs in Data Center Lok >2014-03-27 16:18:22,737 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated templates OVFs in Data Center Lok >2014-03-27 16:18:22,737 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 16:18:22,738 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 16:23:22,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1e399324 >2014-03-27 16:23:22,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 55eb2ffc >2014-03-27 16:23:22,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 55eb2ffc >2014-03-27 16:23:22,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1e399324 >2014-03-27 16:23:22,839 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-27 17:18:22,487 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-6) Start refreshing all users data >2014-03-27 17:18:22,739 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-03-27 17:18:22,740 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Lok >2014-03-27 17:18:22,741 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Lok >2014-03-27 17:18:22,741 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Lok >2014-03-27 17:18:22,742 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated templates OVFs in Data Center Lok >2014-03-27 17:18:22,742 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 17:18:22,743 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 17:23:22,844 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6528a989 >2014-03-27 17:23:22,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 50929541 >2014-03-27 17:23:23,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 50929541 >2014-03-27 17:23:23,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6528a989 >2014-03-27 17:23:23,085 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-27 18:18:22,489 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) [6523099e] Start refreshing all users data >2014-03-27 18:18:22,743 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-03-27 18:18:22,744 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VM OVFs in Data Center Lok >2014-03-27 18:18:22,745 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated VM OVFs in Data Center Lok >2014-03-27 18:18:22,745 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update template OVFs in Data Center Lok >2014-03-27 18:18:22,747 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated templates OVFs in Data Center Lok >2014-03-27 18:18:22,747 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 18:18:22,748 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 18:23:23,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3a903343 >2014-03-27 18:23:23,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 511591be >2014-03-27 18:23:23,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 511591be >2014-03-27 18:23:23,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3a903343 >2014-03-27 18:23:23,334 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-27 19:18:22,491 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-88) Start refreshing all users data >2014-03-27 19:18:22,748 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-03-27 19:18:22,749 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VM OVFs in Data Center Lok >2014-03-27 19:18:22,750 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated VM OVFs in Data Center Lok >2014-03-27 19:18:22,750 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update template OVFs in Data Center Lok >2014-03-27 19:18:22,751 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated templates OVFs in Data Center Lok >2014-03-27 19:18:22,751 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 19:18:22,752 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 19:23:23,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) [6523099e] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3c57ddef >2014-03-27 19:23:23,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) [6523099e] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2cab94dc >2014-03-27 19:23:23,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) [6523099e] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2cab94dc >2014-03-27 19:23:23,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) [6523099e] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c57ddef >2014-03-27 19:23:23,570 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) [6523099e] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-27 20:18:22,493 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-03-27 20:18:22,752 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-03-27 20:18:22,753 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Lok >2014-03-27 20:18:22,755 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Lok >2014-03-27 20:18:22,755 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Lok >2014-03-27 20:18:22,756 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated templates OVFs in Data Center Lok >2014-03-27 20:18:22,756 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 20:18:22,757 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 20:23:23,576 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 53f5e7c6 >2014-03-27 20:23:23,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3543a33a >2014-03-27 20:23:23,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3543a33a >2014-03-27 20:23:23,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 53f5e7c6 >2014-03-27 20:23:23,815 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-27 21:18:22,496 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-03-27 21:18:22,757 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-03-27 21:18:22,758 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VM OVFs in Data Center Lok >2014-03-27 21:18:22,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated VM OVFs in Data Center Lok >2014-03-27 21:18:22,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update template OVFs in Data Center Lok >2014-03-27 21:18:22,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated templates OVFs in Data Center Lok >2014-03-27 21:18:22,763 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 21:18:22,763 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 21:23:23,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6c437ecf >2014-03-27 21:23:23,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 433a7f90 >2014-03-27 21:23:24,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 433a7f90 >2014-03-27 21:23:24,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6c437ecf >2014-03-27 21:23:24,055 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-27 22:18:22,498 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-03-27 22:18:22,763 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-03-27 22:18:22,764 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-03-27 22:18:22,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-03-27 22:18:22,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-03-27 22:18:22,766 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-03-27 22:18:22,766 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 22:18:22,766 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 22:23:24,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3e88402e >2014-03-27 22:23:24,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 632dac14 >2014-03-27 22:23:24,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 632dac14 >2014-03-27 22:23:24,286 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3e88402e >2014-03-27 22:23:24,292 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-27 23:18:22,499 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-03-27 23:18:22,766 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-03-27 23:18:22,767 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-03-27 23:18:22,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-03-27 23:18:22,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-03-27 23:18:22,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-03-27 23:18:22,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-27 23:18:22,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-27 23:23:24,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2383f01f >2014-03-27 23:23:24,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2b099570 >2014-03-27 23:23:24,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2b099570 >2014-03-27 23:23:24,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2383f01f >2014-03-27 23:23:24,534 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 00:18:22,501 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-03-28 00:18:22,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-03-28 00:18:22,770 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VM OVFs in Data Center Lok >2014-03-28 00:18:22,771 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Successfully updated VM OVFs in Data Center Lok >2014-03-28 00:18:22,771 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update template OVFs in Data Center Lok >2014-03-28 00:18:22,772 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Successfully updated templates OVFs in Data Center Lok >2014-03-28 00:18:22,772 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 00:18:22,773 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 00:23:24,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4ce30b44 >2014-03-28 00:23:24,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4df8c4cb >2014-03-28 00:23:24,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4df8c4cb >2014-03-28 00:23:24,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4ce30b44 >2014-03-28 00:23:24,779 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 01:18:22,503 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-03-28 01:18:22,773 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-03-28 01:18:22,774 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-03-28 01:18:22,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-03-28 01:18:22,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-03-28 01:18:22,776 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-03-28 01:18:22,776 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 01:18:22,776 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 01:23:24,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d943493 >2014-03-28 01:23:24,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6d07ae10 >2014-03-28 01:23:25,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6d07ae10 >2014-03-28 01:23:25,015 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d943493 >2014-03-28 01:23:25,023 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 02:18:22,505 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-03-28 02:18:22,777 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-03-28 02:18:22,778 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VM OVFs in Data Center Lok >2014-03-28 02:18:22,783 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated VM OVFs in Data Center Lok >2014-03-28 02:18:22,783 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update template OVFs in Data Center Lok >2014-03-28 02:18:22,783 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated templates OVFs in Data Center Lok >2014-03-28 02:18:22,783 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 02:18:22,784 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 02:23:25,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2b3671f1 >2014-03-28 02:23:25,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3729b61 >2014-03-28 02:23:25,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3729b61 >2014-03-28 02:23:25,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2b3671f1 >2014-03-28 02:23:25,268 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 03:18:22,508 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-03-28 03:18:22,784 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-03-28 03:18:22,785 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VM OVFs in Data Center Lok >2014-03-28 03:18:22,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated VM OVFs in Data Center Lok >2014-03-28 03:18:22,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update template OVFs in Data Center Lok >2014-03-28 03:18:22,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated templates OVFs in Data Center Lok >2014-03-28 03:18:22,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 03:18:22,787 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 03:23:25,272 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: cc59a18 >2014-03-28 03:23:25,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6086bcc7 >2014-03-28 03:23:25,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6086bcc7 >2014-03-28 03:23:25,569 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: cc59a18 >2014-03-28 03:23:25,584 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-36) Start deleteAgedOutAuditLogs >2014-03-28 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-36) Finished deleteAgedOutAuditLogs >2014-03-28 04:18:22,509 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-03-28 04:18:22,787 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-03-28 04:18:22,788 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-03-28 04:18:22,788 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-03-28 04:18:22,788 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-03-28 04:18:22,789 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-03-28 04:18:22,789 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 04:18:22,789 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 04:23:25,587 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3c43c91 >2014-03-28 04:23:25,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 48645585 >2014-03-28 04:23:25,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 48645585 >2014-03-28 04:23:25,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c43c91 >2014-03-28 04:23:25,826 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 05:18:22,511 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-88) Start refreshing all users data >2014-03-28 05:18:22,789 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-03-28 05:18:22,790 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-03-28 05:18:22,790 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-03-28 05:18:22,791 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-03-28 05:18:22,791 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-03-28 05:18:22,791 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 05:18:22,792 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 05:23:25,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7c9e660 >2014-03-28 05:23:25,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5833ac9e >2014-03-28 05:23:26,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5833ac9e >2014-03-28 05:23:26,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c9e660 >2014-03-28 05:23:26,073 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 06:18:22,512 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-03-28 06:18:22,792 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-03-28 06:18:22,793 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VM OVFs in Data Center Lok >2014-03-28 06:18:22,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated VM OVFs in Data Center Lok >2014-03-28 06:18:22,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update template OVFs in Data Center Lok >2014-03-28 06:18:22,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated templates OVFs in Data Center Lok >2014-03-28 06:18:22,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 06:18:22,795 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 06:23:26,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5e9a9fe4 >2014-03-28 06:23:26,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 79776241 >2014-03-28 06:23:26,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 79776241 >2014-03-28 06:23:26,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e9a9fe4 >2014-03-28 06:23:26,316 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 07:18:22,514 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-03-28 07:18:22,795 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-03-28 07:18:22,796 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Lok >2014-03-28 07:18:22,796 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Lok >2014-03-28 07:18:22,797 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Lok >2014-03-28 07:18:22,797 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center Lok >2014-03-28 07:18:22,797 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 07:18:22,798 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 07:23:26,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 634a7ab >2014-03-28 07:23:26,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6aebcebb >2014-03-28 07:23:26,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6aebcebb >2014-03-28 07:23:26,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 634a7ab >2014-03-28 07:23:26,557 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 08:18:22,516 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-2) Start refreshing all users data >2014-03-28 08:18:22,798 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-03-28 08:18:22,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-03-28 08:18:22,800 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-03-28 08:18:22,800 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-03-28 08:18:22,801 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-03-28 08:18:22,801 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 08:18:22,801 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 08:23:26,559 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4f0e4e44 >2014-03-28 08:23:26,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1d9a4f66 >2014-03-28 08:23:27,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d9a4f66 >2014-03-28 08:23:27,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f0e4e44 >2014-03-28 08:23:27,024 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 09:18:22,517 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-03-28 09:18:22,801 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-03-28 09:18:22,802 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VM OVFs in Data Center Lok >2014-03-28 09:18:22,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Successfully updated VM OVFs in Data Center Lok >2014-03-28 09:18:22,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update template OVFs in Data Center Lok >2014-03-28 09:18:22,806 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Successfully updated templates OVFs in Data Center Lok >2014-03-28 09:18:22,806 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 09:18:22,806 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 09:23:27,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 318be18f >2014-03-28 09:23:27,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 56858cb2 >2014-03-28 09:23:27,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 56858cb2 >2014-03-28 09:23:27,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 318be18f >2014-03-28 09:23:27,264 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 10:18:22,518 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-03-28 10:18:22,806 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-03-28 10:18:22,807 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VM OVFs in Data Center Lok >2014-03-28 10:18:22,808 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated VM OVFs in Data Center Lok >2014-03-28 10:18:22,808 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update template OVFs in Data Center Lok >2014-03-28 10:18:22,809 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated templates OVFs in Data Center Lok >2014-03-28 10:18:22,809 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 10:18:22,809 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 10:23:27,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 258e1f01 >2014-03-28 10:23:27,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 15958dea >2014-03-28 10:23:27,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 15958dea >2014-03-28 10:23:27,673 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 258e1f01 >2014-03-28 10:23:27,679 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 11:18:22,520 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-03-28 11:18:22,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-03-28 11:18:22,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-03-28 11:18:22,811 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-03-28 11:18:22,811 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-03-28 11:18:22,812 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-03-28 11:18:22,812 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 11:18:22,812 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 11:23:27,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 470ebc88 >2014-03-28 11:23:27,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 187ba92a >2014-03-28 11:23:27,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 187ba92a >2014-03-28 11:23:27,918 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 470ebc88 >2014-03-28 11:23:27,938 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 12:18:22,522 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-03-28 12:18:22,813 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-03-28 12:18:22,814 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VM OVFs in Data Center Lok >2014-03-28 12:18:22,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated VM OVFs in Data Center Lok >2014-03-28 12:18:22,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update template OVFs in Data Center Lok >2014-03-28 12:18:22,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated templates OVFs in Data Center Lok >2014-03-28 12:18:22,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 12:18:22,816 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 12:23:27,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 64755300 >2014-03-28 12:23:27,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 529c5886 >2014-03-28 12:23:28,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 529c5886 >2014-03-28 12:23:28,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 64755300 >2014-03-28 12:23:28,172 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 13:18:22,524 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-03-28 13:18:22,816 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-03-28 13:18:22,817 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VM OVFs in Data Center Lok >2014-03-28 13:18:22,817 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully updated VM OVFs in Data Center Lok >2014-03-28 13:18:22,817 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update template OVFs in Data Center Lok >2014-03-28 13:18:22,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully updated templates OVFs in Data Center Lok >2014-03-28 13:18:22,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 13:18:22,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 13:23:28,174 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 37311ce7 >2014-03-28 13:23:28,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 79609dff >2014-03-28 13:23:28,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 79609dff >2014-03-28 13:23:28,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 37311ce7 >2014-03-28 13:23:28,409 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 14:18:22,526 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-03-28 14:18:22,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-28 14:18:22,819 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-03-28 14:18:22,819 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-03-28 14:18:22,819 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-03-28 14:18:22,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-03-28 14:18:22,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 14:18:22,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 14:23:28,413 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3a056cad >2014-03-28 14:23:28,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3a21ef9d >2014-03-28 14:23:28,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3a21ef9d >2014-03-28 14:23:28,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3a056cad >2014-03-28 14:23:28,645 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 15:18:22,528 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-03-28 15:18:22,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-28 15:18:22,821 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-03-28 15:18:22,821 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-03-28 15:18:22,821 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-03-28 15:18:22,822 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-03-28 15:18:22,822 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 15:18:22,822 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 15:23:28,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2d706395 >2014-03-28 15:23:28,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2d99e281 >2014-03-28 15:23:28,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d99e281 >2014-03-28 15:23:28,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d706395 >2014-03-28 15:23:28,886 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 16:18:22,532 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-28 16:18:22,822 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-03-28 16:18:22,823 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VM OVFs in Data Center Lok >2014-03-28 16:18:22,824 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated VM OVFs in Data Center Lok >2014-03-28 16:18:22,824 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update template OVFs in Data Center Lok >2014-03-28 16:18:22,825 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated templates OVFs in Data Center Lok >2014-03-28 16:18:22,825 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 16:18:22,825 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 16:23:28,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 65fb2c29 >2014-03-28 16:23:28,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4d106c0d >2014-03-28 16:23:29,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4d106c0d >2014-03-28 16:23:29,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 65fb2c29 >2014-03-28 16:23:29,125 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 17:18:22,534 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) Start refreshing all users data >2014-03-28 17:18:22,825 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-03-28 17:18:22,826 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VM OVFs in Data Center Lok >2014-03-28 17:18:22,827 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated VM OVFs in Data Center Lok >2014-03-28 17:18:22,827 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update template OVFs in Data Center Lok >2014-03-28 17:18:22,827 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated templates OVFs in Data Center Lok >2014-03-28 17:18:22,827 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 17:18:22,828 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 17:23:29,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 64454c33 >2014-03-28 17:23:29,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 18ddbb3 >2014-03-28 17:23:29,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 18ddbb3 >2014-03-28 17:23:29,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 64454c33 >2014-03-28 17:23:29,366 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 18:18:22,536 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) Start refreshing all users data >2014-03-28 18:18:22,828 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-03-28 18:18:22,829 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Lok >2014-03-28 18:18:22,830 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Lok >2014-03-28 18:18:22,830 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Lok >2014-03-28 18:18:22,831 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center Lok >2014-03-28 18:18:22,831 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 18:18:22,831 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 18:23:29,370 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 551515e6 >2014-03-28 18:23:29,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5b50daad >2014-03-28 18:23:29,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5b50daad >2014-03-28 18:23:29,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 551515e6 >2014-03-28 18:23:29,607 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 19:18:22,538 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-03-28 19:18:22,832 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-03-28 19:18:22,833 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VM OVFs in Data Center Lok >2014-03-28 19:18:22,833 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated VM OVFs in Data Center Lok >2014-03-28 19:18:22,833 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update template OVFs in Data Center Lok >2014-03-28 19:18:22,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated templates OVFs in Data Center Lok >2014-03-28 19:18:22,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 19:18:22,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 19:23:29,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 45e3a7a7 >2014-03-28 19:23:29,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 18ff16aa >2014-03-28 19:23:30,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 18ff16aa >2014-03-28 19:23:30,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 45e3a7a7 >2014-03-28 19:23:30,203 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 20:18:22,539 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-03-28 20:18:22,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-28 20:18:22,835 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-03-28 20:18:22,836 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-03-28 20:18:22,836 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-03-28 20:18:22,837 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-03-28 20:18:22,837 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 20:18:22,837 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 20:23:30,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 60545843 >2014-03-28 20:23:30,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 416ec81d >2014-03-28 20:23:30,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 416ec81d >2014-03-28 20:23:30,737 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 60545843 >2014-03-28 20:23:30,743 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 21:18:22,541 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-28 21:18:22,838 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-03-28 21:18:22,838 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VM OVFs in Data Center Lok >2014-03-28 21:18:22,839 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated VM OVFs in Data Center Lok >2014-03-28 21:18:22,839 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update template OVFs in Data Center Lok >2014-03-28 21:18:22,840 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated templates OVFs in Data Center Lok >2014-03-28 21:18:22,840 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 21:18:22,840 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 21:23:30,750 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 45a7b7e9 >2014-03-28 21:23:30,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 61b45b9 >2014-03-28 21:23:30,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 61b45b9 >2014-03-28 21:23:30,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 45a7b7e9 >2014-03-28 21:23:30,986 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 22:18:22,543 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-03-28 22:18:22,841 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-03-28 22:18:22,842 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-03-28 22:18:22,842 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-03-28 22:18:22,842 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-03-28 22:18:22,843 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-03-28 22:18:22,843 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 22:18:22,844 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 22:23:30,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7ab5d8fe >2014-03-28 22:23:30,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 38b94aa2 >2014-03-28 22:23:31,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 38b94aa2 >2014-03-28 22:23:31,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7ab5d8fe >2014-03-28 22:23:31,222 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-28 23:18:22,546 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-03-28 23:18:22,844 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-03-28 23:18:22,845 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VM OVFs in Data Center Lok >2014-03-28 23:18:22,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated VM OVFs in Data Center Lok >2014-03-28 23:18:22,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update template OVFs in Data Center Lok >2014-03-28 23:18:22,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated templates OVFs in Data Center Lok >2014-03-28 23:18:22,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-28 23:18:22,847 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-28 23:23:31,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4aea0529 >2014-03-28 23:23:31,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 49f8ed2b >2014-03-28 23:23:31,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 49f8ed2b >2014-03-28 23:23:31,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4aea0529 >2014-03-28 23:23:31,466 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 00:18:22,548 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-03-29 00:18:22,847 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-03-29 00:18:22,848 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VM OVFs in Data Center Lok >2014-03-29 00:18:22,849 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated VM OVFs in Data Center Lok >2014-03-29 00:18:22,849 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update template OVFs in Data Center Lok >2014-03-29 00:18:22,850 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated templates OVFs in Data Center Lok >2014-03-29 00:18:22,850 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 00:18:22,851 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 00:23:31,469 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2da0e83b >2014-03-29 00:23:31,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 307f75c2 >2014-03-29 00:23:31,889 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 307f75c2 >2014-03-29 00:23:31,889 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2da0e83b >2014-03-29 00:23:31,894 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 01:18:22,550 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-03-29 01:18:22,851 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-03-29 01:18:22,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VM OVFs in Data Center Lok >2014-03-29 01:18:22,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated VM OVFs in Data Center Lok >2014-03-29 01:18:22,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update template OVFs in Data Center Lok >2014-03-29 01:18:22,854 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated templates OVFs in Data Center Lok >2014-03-29 01:18:22,854 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 01:18:22,855 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 01:23:31,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7c6ac8af >2014-03-29 01:23:31,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 469d98f9 >2014-03-29 01:23:32,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 469d98f9 >2014-03-29 01:23:32,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c6ac8af >2014-03-29 01:23:32,136 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 02:18:22,552 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-03-29 02:18:22,856 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-03-29 02:18:22,857 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VM OVFs in Data Center Lok >2014-03-29 02:18:22,858 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully updated VM OVFs in Data Center Lok >2014-03-29 02:18:22,858 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update template OVFs in Data Center Lok >2014-03-29 02:18:22,858 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully updated templates OVFs in Data Center Lok >2014-03-29 02:18:22,858 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 02:18:22,859 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 02:23:32,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d454501 >2014-03-29 02:23:32,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2eb553aa >2014-03-29 02:23:32,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2eb553aa >2014-03-29 02:23:32,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d454501 >2014-03-29 02:23:32,377 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 03:18:22,553 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-41) Start refreshing all users data >2014-03-29 03:18:22,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-29 03:18:22,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-03-29 03:18:22,861 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-03-29 03:18:22,861 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-03-29 03:18:22,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-03-29 03:18:22,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 03:18:22,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 03:23:32,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7d80a785 >2014-03-29 03:23:32,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 9764d22 >2014-03-29 03:23:32,612 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 9764d22 >2014-03-29 03:23:32,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7d80a785 >2014-03-29 03:23:32,617 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-37) Start deleteAgedOutAuditLogs >2014-03-29 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-37) Finished deleteAgedOutAuditLogs >2014-03-29 04:18:22,555 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-2) Start refreshing all users data >2014-03-29 04:18:22,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-03-29 04:18:22,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VM OVFs in Data Center Lok >2014-03-29 04:18:22,863 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated VM OVFs in Data Center Lok >2014-03-29 04:18:22,863 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update template OVFs in Data Center Lok >2014-03-29 04:18:22,863 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated templates OVFs in Data Center Lok >2014-03-29 04:18:22,863 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 04:18:22,864 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 04:23:32,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 405d4f89 >2014-03-29 04:23:32,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2e10cc62 >2014-03-29 04:23:32,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e10cc62 >2014-03-29 04:23:32,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 405d4f89 >2014-03-29 04:23:32,862 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 05:18:22,557 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-03-29 05:18:22,864 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-03-29 05:18:22,865 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-03-29 05:18:22,866 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-03-29 05:18:22,866 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-03-29 05:18:22,866 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-03-29 05:18:22,866 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 05:18:22,867 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 05:23:32,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1bd56bd0 >2014-03-29 05:23:33,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 23848875 >2014-03-29 05:23:33,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23848875 >2014-03-29 05:23:33,429 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1bd56bd0 >2014-03-29 05:23:33,435 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 06:18:22,559 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-03-29 06:18:22,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-03-29 06:18:22,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VM OVFs in Data Center Lok >2014-03-29 06:18:22,869 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated VM OVFs in Data Center Lok >2014-03-29 06:18:22,869 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update template OVFs in Data Center Lok >2014-03-29 06:18:22,869 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated templates OVFs in Data Center Lok >2014-03-29 06:18:22,869 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 06:18:22,870 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 06:23:33,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5947c377 >2014-03-29 06:23:33,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7671e76b >2014-03-29 06:23:33,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7671e76b >2014-03-29 06:23:33,846 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5947c377 >2014-03-29 06:23:33,850 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 07:18:22,562 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-03-29 07:18:22,870 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-03-29 07:18:22,871 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VM OVFs in Data Center Lok >2014-03-29 07:18:22,872 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated VM OVFs in Data Center Lok >2014-03-29 07:18:22,872 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update template OVFs in Data Center Lok >2014-03-29 07:18:22,872 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated templates OVFs in Data Center Lok >2014-03-29 07:18:22,872 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 07:18:22,873 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 07:23:33,854 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4e830e91 >2014-03-29 07:23:33,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7e3179db >2014-03-29 07:23:34,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7e3179db >2014-03-29 07:23:34,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4e830e91 >2014-03-29 07:23:34,095 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 08:18:22,563 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-03-29 08:18:22,874 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-03-29 08:18:22,874 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-03-29 08:18:22,875 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-03-29 08:18:22,875 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-03-29 08:18:22,876 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-03-29 08:18:22,876 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 08:18:22,876 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 08:23:34,099 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 71e15f7 >2014-03-29 08:23:34,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 48340ec7 >2014-03-29 08:23:34,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 48340ec7 >2014-03-29 08:23:34,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 71e15f7 >2014-03-29 08:23:34,324 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 09:18:22,564 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-03-29 09:18:22,876 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [6523099e] Attempting to update VMs/Templates Ovf. >2014-03-29 09:18:22,877 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [6523099e] Attempting to update VM OVFs in Data Center Lok >2014-03-29 09:18:22,878 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [6523099e] Successfully updated VM OVFs in Data Center Lok >2014-03-29 09:18:22,878 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [6523099e] Attempting to update template OVFs in Data Center Lok >2014-03-29 09:18:22,879 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [6523099e] Successfully updated templates OVFs in Data Center Lok >2014-03-29 09:18:22,879 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [6523099e] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 09:18:22,879 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) [6523099e] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 09:23:34,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6e79b9e0 >2014-03-29 09:23:34,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 799a59c9 >2014-03-29 09:23:34,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 799a59c9 >2014-03-29 09:23:34,548 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6e79b9e0 >2014-03-29 09:23:34,553 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 10:18:22,566 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-03-29 10:18:22,879 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-03-29 10:18:22,879 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VM OVFs in Data Center Lok >2014-03-29 10:18:22,880 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated VM OVFs in Data Center Lok >2014-03-29 10:18:22,880 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update template OVFs in Data Center Lok >2014-03-29 10:18:22,881 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated templates OVFs in Data Center Lok >2014-03-29 10:18:22,881 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 10:18:22,881 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 10:23:34,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 10feb9d9 >2014-03-29 10:23:34,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 435b5e46 >2014-03-29 10:23:34,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 435b5e46 >2014-03-29 10:23:34,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 10feb9d9 >2014-03-29 10:23:34,796 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 11:18:22,568 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-69) Start refreshing all users data >2014-03-29 11:18:22,881 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-03-29 11:18:22,882 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VM OVFs in Data Center Lok >2014-03-29 11:18:22,883 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated VM OVFs in Data Center Lok >2014-03-29 11:18:22,883 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update template OVFs in Data Center Lok >2014-03-29 11:18:22,884 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated templates OVFs in Data Center Lok >2014-03-29 11:18:22,884 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 11:18:22,884 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 11:23:34,800 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7cc25474 >2014-03-29 11:23:34,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3a36cba9 >2014-03-29 11:23:35,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3a36cba9 >2014-03-29 11:23:35,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7cc25474 >2014-03-29 11:23:35,036 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 12:18:22,570 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-03-29 12:18:22,884 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-03-29 12:18:22,884 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VM OVFs in Data Center Lok >2014-03-29 12:18:22,885 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully updated VM OVFs in Data Center Lok >2014-03-29 12:18:22,885 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update template OVFs in Data Center Lok >2014-03-29 12:18:22,885 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully updated templates OVFs in Data Center Lok >2014-03-29 12:18:22,886 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 12:18:22,886 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 12:23:35,038 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1a9d2464 >2014-03-29 12:23:35,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 42c02ee2 >2014-03-29 12:23:35,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 42c02ee2 >2014-03-29 12:23:35,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1a9d2464 >2014-03-29 12:23:35,280 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 13:18:22,570 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-03-29 13:18:22,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-03-29 13:18:22,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VM OVFs in Data Center Lok >2014-03-29 13:18:22,888 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated VM OVFs in Data Center Lok >2014-03-29 13:18:22,888 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update template OVFs in Data Center Lok >2014-03-29 13:18:22,889 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated templates OVFs in Data Center Lok >2014-03-29 13:18:22,889 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 13:18:22,889 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 13:23:35,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 99e2942 >2014-03-29 13:23:35,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6ea6b0fc >2014-03-29 13:23:35,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ea6b0fc >2014-03-29 13:23:35,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 99e2942 >2014-03-29 13:23:35,522 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 14:18:22,573 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-03-29 14:18:22,889 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-03-29 14:18:22,890 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VM OVFs in Data Center Lok >2014-03-29 14:18:22,891 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated VM OVFs in Data Center Lok >2014-03-29 14:18:22,891 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update template OVFs in Data Center Lok >2014-03-29 14:18:22,891 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated templates OVFs in Data Center Lok >2014-03-29 14:18:22,891 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 14:18:22,892 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 14:23:35,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: ff830d0 >2014-03-29 14:23:35,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6f4cafe0 >2014-03-29 14:23:35,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6f4cafe0 >2014-03-29 14:23:35,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ff830d0 >2014-03-29 14:23:35,767 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 15:18:22,575 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-50) Start refreshing all users data >2014-03-29 15:18:22,892 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-29 15:18:22,893 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-03-29 15:18:22,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-03-29 15:18:22,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-03-29 15:18:22,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-03-29 15:18:22,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 15:18:22,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 15:23:35,771 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5151ef83 >2014-03-29 15:23:35,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5a43ab2a >2014-03-29 15:23:36,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a43ab2a >2014-03-29 15:23:36,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5151ef83 >2014-03-29 15:23:36,007 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 16:18:22,577 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-41) Start refreshing all users data >2014-03-29 16:18:22,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-03-29 16:18:22,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-03-29 16:18:22,897 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-03-29 16:18:22,897 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-03-29 16:18:22,897 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-03-29 16:18:22,897 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 16:18:22,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 16:23:36,012 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5384a1ab >2014-03-29 16:23:36,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 28920c63 >2014-03-29 16:23:36,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 28920c63 >2014-03-29 16:23:36,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5384a1ab >2014-03-29 16:23:36,251 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 17:18:22,579 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-03-29 17:18:22,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-29 17:18:22,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VM OVFs in Data Center Lok >2014-03-29 17:18:22,899 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Successfully updated VM OVFs in Data Center Lok >2014-03-29 17:18:22,899 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update template OVFs in Data Center Lok >2014-03-29 17:18:22,900 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Successfully updated templates OVFs in Data Center Lok >2014-03-29 17:18:22,900 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 17:18:22,900 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 17:23:36,254 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3b39f1a6 >2014-03-29 17:23:36,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5927e7aa >2014-03-29 17:23:36,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5927e7aa >2014-03-29 17:23:36,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3b39f1a6 >2014-03-29 17:23:36,495 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 18:18:22,580 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-03-29 18:18:22,901 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-03-29 18:18:22,902 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VM OVFs in Data Center Lok >2014-03-29 18:18:22,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated VM OVFs in Data Center Lok >2014-03-29 18:18:22,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update template OVFs in Data Center Lok >2014-03-29 18:18:22,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated templates OVFs in Data Center Lok >2014-03-29 18:18:22,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 18:18:22,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 18:23:36,499 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 35bd67d >2014-03-29 18:23:36,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 25eaa44 >2014-03-29 18:23:36,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 25eaa44 >2014-03-29 18:23:36,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 35bd67d >2014-03-29 18:23:36,740 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 19:18:22,583 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-03-29 19:18:22,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-03-29 19:18:22,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VM OVFs in Data Center Lok >2014-03-29 19:18:22,907 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated VM OVFs in Data Center Lok >2014-03-29 19:18:22,907 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update template OVFs in Data Center Lok >2014-03-29 19:18:22,908 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated templates OVFs in Data Center Lok >2014-03-29 19:18:22,908 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 19:18:22,908 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 19:23:36,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 50002491 >2014-03-29 19:23:36,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3960d49f >2014-03-29 19:23:36,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3960d49f >2014-03-29 19:23:36,971 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 50002491 >2014-03-29 19:23:36,977 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 20:18:22,585 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-03-29 20:18:22,909 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-03-29 20:18:22,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VM OVFs in Data Center Lok >2014-03-29 20:18:22,911 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated VM OVFs in Data Center Lok >2014-03-29 20:18:22,911 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update template OVFs in Data Center Lok >2014-03-29 20:18:22,911 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated templates OVFs in Data Center Lok >2014-03-29 20:18:22,911 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 20:18:22,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 20:23:36,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6bbae9ec >2014-03-29 20:23:36,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3c53bc35 >2014-03-29 20:23:37,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c53bc35 >2014-03-29 20:23:37,503 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6bbae9ec >2014-03-29 20:23:37,509 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 21:18:22,586 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-03-29 21:18:22,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-03-29 21:18:22,913 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VM OVFs in Data Center Lok >2014-03-29 21:18:22,914 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated VM OVFs in Data Center Lok >2014-03-29 21:18:22,914 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update template OVFs in Data Center Lok >2014-03-29 21:18:22,914 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated templates OVFs in Data Center Lok >2014-03-29 21:18:22,914 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 21:18:22,915 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 21:23:37,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5fdf8333 >2014-03-29 21:23:37,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4b1eba71 >2014-03-29 21:23:38,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b1eba71 >2014-03-29 21:23:38,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5fdf8333 >2014-03-29 21:23:38,023 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 22:18:22,588 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-03-29 22:18:22,916 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-03-29 22:18:22,917 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VM OVFs in Data Center Lok >2014-03-29 22:18:22,917 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated VM OVFs in Data Center Lok >2014-03-29 22:18:22,917 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update template OVFs in Data Center Lok >2014-03-29 22:18:22,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated templates OVFs in Data Center Lok >2014-03-29 22:18:22,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 22:18:22,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 22:23:38,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 73a8fa02 >2014-03-29 22:23:38,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5ab5d4a0 >2014-03-29 22:23:38,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5ab5d4a0 >2014-03-29 22:23:38,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 73a8fa02 >2014-03-29 22:23:38,266 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-29 23:18:22,589 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-03-29 23:18:22,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-03-29 23:18:22,919 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VM OVFs in Data Center Lok >2014-03-29 23:18:22,920 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated VM OVFs in Data Center Lok >2014-03-29 23:18:22,920 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update template OVFs in Data Center Lok >2014-03-29 23:18:22,921 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated templates OVFs in Data Center Lok >2014-03-29 23:18:22,921 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-29 23:18:22,921 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-29 23:23:38,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3748a68b >2014-03-29 23:23:38,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2ae0c08 >2014-03-29 23:23:38,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2ae0c08 >2014-03-29 23:23:38,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3748a68b >2014-03-29 23:23:38,507 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 00:18:22,591 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-03-30 00:18:22,921 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-03-30 00:18:22,922 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VM OVFs in Data Center Lok >2014-03-30 00:18:22,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated VM OVFs in Data Center Lok >2014-03-30 00:18:22,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update template OVFs in Data Center Lok >2014-03-30 00:18:22,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated templates OVFs in Data Center Lok >2014-03-30 00:18:22,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 00:18:22,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 00:23:38,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 208c6295 >2014-03-30 00:23:38,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 609c34e4 >2014-03-30 00:23:38,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 609c34e4 >2014-03-30 00:23:38,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 208c6295 >2014-03-30 00:23:38,749 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 01:18:22,593 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-03-30 01:18:22,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-03-30 01:18:22,925 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Lok >2014-03-30 01:18:22,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Lok >2014-03-30 01:18:22,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Lok >2014-03-30 01:18:22,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center Lok >2014-03-30 01:18:22,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 01:18:22,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 01:23:38,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 40bdf7 >2014-03-30 01:23:38,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1a77a342 >2014-03-30 01:23:38,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1a77a342 >2014-03-30 01:23:38,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 40bdf7 >2014-03-30 01:23:38,991 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 03:18:22,593 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-03-30 03:18:22,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-03-30 03:18:22,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-03-30 03:18:22,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-03-30 03:18:22,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-03-30 03:18:22,929 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-03-30 03:18:22,929 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 03:18:22,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 03:23:38,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 59777beb >2014-03-30 03:23:38,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3368af30 >2014-03-30 03:23:39,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3368af30 >2014-03-30 03:23:39,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 59777beb >2014-03-30 03:23:39,232 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-33) Start deleteAgedOutAuditLogs >2014-03-30 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-33) Finished deleteAgedOutAuditLogs >2014-03-30 04:18:22,595 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-03-30 04:18:22,931 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-03-30 04:18:22,931 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VM OVFs in Data Center Lok >2014-03-30 04:18:22,932 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated VM OVFs in Data Center Lok >2014-03-30 04:18:22,932 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update template OVFs in Data Center Lok >2014-03-30 04:18:22,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated templates OVFs in Data Center Lok >2014-03-30 04:18:22,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 04:18:22,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 04:23:39,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1ab1dc5d >2014-03-30 04:23:39,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 12fa6b5c >2014-03-30 04:23:39,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 12fa6b5c >2014-03-30 04:23:39,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1ab1dc5d >2014-03-30 04:23:39,469 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 05:18:22,597 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-03-30 05:18:22,934 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-03-30 05:18:22,934 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VM OVFs in Data Center Lok >2014-03-30 05:18:22,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated VM OVFs in Data Center Lok >2014-03-30 05:18:22,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update template OVFs in Data Center Lok >2014-03-30 05:18:22,936 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated templates OVFs in Data Center Lok >2014-03-30 05:18:22,936 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 05:18:22,936 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 05:23:39,474 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 62736622 >2014-03-30 05:23:39,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 63674ad >2014-03-30 05:23:39,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 63674ad >2014-03-30 05:23:39,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 62736622 >2014-03-30 05:23:39,709 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 06:18:22,599 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-03-30 06:18:22,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-03-30 06:18:22,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-03-30 06:18:22,939 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-03-30 06:18:22,939 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-03-30 06:18:22,940 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-03-30 06:18:22,940 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 06:18:22,940 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 06:23:39,717 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 74e19e2f >2014-03-30 06:23:39,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 8585b9c >2014-03-30 06:23:39,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 8585b9c >2014-03-30 06:23:39,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 74e19e2f >2014-03-30 06:23:39,953 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 07:18:22,602 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-03-30 07:18:22,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-03-30 07:18:22,942 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-03-30 07:18:22,943 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-03-30 07:18:22,943 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-03-30 07:18:22,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-03-30 07:18:22,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 07:18:22,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 07:23:39,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 305fd1b8 >2014-03-30 07:23:39,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 62c19fd7 >2014-03-30 07:23:40,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 62c19fd7 >2014-03-30 07:23:40,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 305fd1b8 >2014-03-30 07:23:40,190 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 08:18:22,604 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-03-30 08:18:22,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-03-30 08:18:22,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VM OVFs in Data Center Lok >2014-03-30 08:18:22,946 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated VM OVFs in Data Center Lok >2014-03-30 08:18:22,946 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update template OVFs in Data Center Lok >2014-03-30 08:18:22,947 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated templates OVFs in Data Center Lok >2014-03-30 08:18:22,947 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 08:18:22,948 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 08:23:40,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 56700583 >2014-03-30 08:23:40,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 547a07ca >2014-03-30 08:23:40,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 547a07ca >2014-03-30 08:23:40,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 56700583 >2014-03-30 08:23:40,432 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 09:18:22,605 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-03-30 09:18:22,949 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-03-30 09:18:22,949 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VM OVFs in Data Center Lok >2014-03-30 09:18:22,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated VM OVFs in Data Center Lok >2014-03-30 09:18:22,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update template OVFs in Data Center Lok >2014-03-30 09:18:22,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated templates OVFs in Data Center Lok >2014-03-30 09:18:22,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 09:18:22,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 09:23:40,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7e04c6d0 >2014-03-30 09:23:40,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6b98c252 >2014-03-30 09:23:40,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6b98c252 >2014-03-30 09:23:40,667 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7e04c6d0 >2014-03-30 09:23:40,672 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 10:18:22,607 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-88) Start refreshing all users data >2014-03-30 10:18:22,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-03-30 10:18:22,952 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VM OVFs in Data Center Lok >2014-03-30 10:18:22,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated VM OVFs in Data Center Lok >2014-03-30 10:18:22,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update template OVFs in Data Center Lok >2014-03-30 10:18:22,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated templates OVFs in Data Center Lok >2014-03-30 10:18:22,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 10:18:22,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 10:23:40,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 346a8014 >2014-03-30 10:23:40,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3bfebbee >2014-03-30 10:23:40,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3bfebbee >2014-03-30 10:23:40,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 346a8014 >2014-03-30 10:23:40,911 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 11:18:22,609 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-03-30 11:18:22,955 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VMs/Templates Ovf. >2014-03-30 11:18:22,956 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update VM OVFs in Data Center Lok >2014-03-30 11:18:22,957 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Successfully updated VM OVFs in Data Center Lok >2014-03-30 11:18:22,957 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to update template OVFs in Data Center Lok >2014-03-30 11:18:22,958 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Successfully updated templates OVFs in Data Center Lok >2014-03-30 11:18:22,958 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 11:18:22,958 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 11:23:40,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 76499a62 >2014-03-30 11:23:40,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5a36d499 >2014-03-30 11:23:41,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a36d499 >2014-03-30 11:23:41,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76499a62 >2014-03-30 11:23:41,147 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 12:18:22,610 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-03-30 12:18:22,958 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-03-30 12:18:22,959 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VM OVFs in Data Center Lok >2014-03-30 12:18:22,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated VM OVFs in Data Center Lok >2014-03-30 12:18:22,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update template OVFs in Data Center Lok >2014-03-30 12:18:22,961 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated templates OVFs in Data Center Lok >2014-03-30 12:18:22,961 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 12:18:22,961 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 12:23:41,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5abb7fa3 >2014-03-30 12:23:41,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4bb8d4de >2014-03-30 12:23:41,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4bb8d4de >2014-03-30 12:23:41,381 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5abb7fa3 >2014-03-30 12:23:41,385 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 13:18:22,612 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-03-30 13:18:22,961 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-03-30 13:18:22,962 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VM OVFs in Data Center Lok >2014-03-30 13:18:22,963 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated VM OVFs in Data Center Lok >2014-03-30 13:18:22,963 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update template OVFs in Data Center Lok >2014-03-30 13:18:22,964 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated templates OVFs in Data Center Lok >2014-03-30 13:18:22,964 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 13:18:22,964 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 13:23:41,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 553363ba >2014-03-30 13:23:41,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 20d87409 >2014-03-30 13:23:41,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 20d87409 >2014-03-30 13:23:41,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 553363ba >2014-03-30 13:23:41,802 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 14:18:22,614 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-03-30 14:18:22,964 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-03-30 14:18:22,965 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-03-30 14:18:22,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-03-30 14:18:22,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-03-30 14:18:22,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-03-30 14:18:22,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 14:18:22,967 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 14:23:41,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 18eddb4b >2014-03-30 14:23:41,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 27d825b6 >2014-03-30 14:23:42,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 27d825b6 >2014-03-30 14:23:42,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 18eddb4b >2014-03-30 14:23:42,039 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 15:18:22,615 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-03-30 15:18:22,968 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-03-30 15:18:22,968 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-03-30 15:18:22,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-03-30 15:18:22,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-03-30 15:18:22,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-03-30 15:18:22,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 15:18:22,970 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 15:23:42,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 24c5e285 >2014-03-30 15:23:42,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3b9a9c9f >2014-03-30 15:23:42,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3b9a9c9f >2014-03-30 15:23:42,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 24c5e285 >2014-03-30 15:23:42,276 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 16:18:22,617 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-03-30 16:18:22,971 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-03-30 16:18:22,971 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Lok >2014-03-30 16:18:22,972 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Lok >2014-03-30 16:18:22,972 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Lok >2014-03-30 16:18:22,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated templates OVFs in Data Center Lok >2014-03-30 16:18:22,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 16:18:22,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 16:23:42,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2ed2408c >2014-03-30 16:23:42,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6d8bbf57 >2014-03-30 16:23:42,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6d8bbf57 >2014-03-30 16:23:42,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2ed2408c >2014-03-30 16:23:42,514 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 17:18:22,619 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-03-30 17:18:22,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-03-30 17:18:22,974 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VM OVFs in Data Center Lok >2014-03-30 17:18:22,975 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated VM OVFs in Data Center Lok >2014-03-30 17:18:22,975 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update template OVFs in Data Center Lok >2014-03-30 17:18:22,975 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated templates OVFs in Data Center Lok >2014-03-30 17:18:22,975 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 17:18:22,976 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 17:23:42,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 354b3750 >2014-03-30 17:23:42,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5d1c8085 >2014-03-30 17:23:42,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5d1c8085 >2014-03-30 17:23:42,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 354b3750 >2014-03-30 17:23:42,758 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 18:18:22,621 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-81) Start refreshing all users data >2014-03-30 18:18:22,976 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-03-30 18:18:22,977 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VM OVFs in Data Center Lok >2014-03-30 18:18:22,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated VM OVFs in Data Center Lok >2014-03-30 18:18:22,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update template OVFs in Data Center Lok >2014-03-30 18:18:22,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated templates OVFs in Data Center Lok >2014-03-30 18:18:22,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 18:18:22,979 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 18:23:42,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2b6ba666 >2014-03-30 18:23:42,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 287988cf >2014-03-30 18:23:42,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 287988cf >2014-03-30 18:23:42,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2b6ba666 >2014-03-30 18:23:43,001 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 19:18:22,622 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-03-30 19:18:22,979 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-03-30 19:18:22,980 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VM OVFs in Data Center Lok >2014-03-30 19:18:22,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully updated VM OVFs in Data Center Lok >2014-03-30 19:18:22,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update template OVFs in Data Center Lok >2014-03-30 19:18:22,982 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully updated templates OVFs in Data Center Lok >2014-03-30 19:18:22,982 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 19:18:22,982 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 19:23:43,004 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 434626d5 >2014-03-30 19:23:43,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 329f8c74 >2014-03-30 19:23:43,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 329f8c74 >2014-03-30 19:23:43,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 434626d5 >2014-03-30 19:23:43,255 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 20:18:22,624 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-03-30 20:18:22,982 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-03-30 20:18:22,984 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-03-30 20:18:22,984 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-03-30 20:18:22,984 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-03-30 20:18:22,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-03-30 20:18:22,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 20:18:22,986 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 20:23:43,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7645d35c >2014-03-30 20:23:43,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 623779ad >2014-03-30 20:23:43,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 623779ad >2014-03-30 20:23:43,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7645d35c >2014-03-30 20:23:43,494 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 21:18:22,626 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-03-30 21:18:22,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-03-30 21:18:22,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VM OVFs in Data Center Lok >2014-03-30 21:18:22,988 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Successfully updated VM OVFs in Data Center Lok >2014-03-30 21:18:22,988 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update template OVFs in Data Center Lok >2014-03-30 21:18:22,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Successfully updated templates OVFs in Data Center Lok >2014-03-30 21:18:22,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 21:18:22,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 21:23:43,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 8eb2e1a >2014-03-30 21:23:43,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 81fb189 >2014-03-30 21:23:43,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 81fb189 >2014-03-30 21:23:43,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 8eb2e1a >2014-03-30 21:23:43,732 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 22:18:22,628 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) Start refreshing all users data >2014-03-30 22:18:22,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-03-30 22:18:22,991 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VM OVFs in Data Center Lok >2014-03-30 22:18:22,992 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated VM OVFs in Data Center Lok >2014-03-30 22:18:22,992 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update template OVFs in Data Center Lok >2014-03-30 22:18:22,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated templates OVFs in Data Center Lok >2014-03-30 22:18:22,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 22:18:22,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 22:23:43,736 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 359ee864 >2014-03-30 22:23:43,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 53ab18bc >2014-03-30 22:23:43,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 53ab18bc >2014-03-30 22:23:43,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 359ee864 >2014-03-30 22:23:43,973 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-30 23:18:22,630 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-30) Start refreshing all users data >2014-03-30 23:18:22,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-03-30 23:18:22,996 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VM OVFs in Data Center Lok >2014-03-30 23:18:22,997 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated VM OVFs in Data Center Lok >2014-03-30 23:18:22,997 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update template OVFs in Data Center Lok >2014-03-30 23:18:22,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated templates OVFs in Data Center Lok >2014-03-30 23:18:22,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-30 23:18:22,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-30 23:23:43,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3246aa68 >2014-03-30 23:23:43,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3d5daba1 >2014-03-30 23:23:44,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3d5daba1 >2014-03-30 23:23:44,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3246aa68 >2014-03-30 23:23:44,209 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 00:18:22,632 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-03-31 00:18:22,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-03-31 00:18:23,000 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VM OVFs in Data Center Lok >2014-03-31 00:18:23,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully updated VM OVFs in Data Center Lok >2014-03-31 00:18:23,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update template OVFs in Data Center Lok >2014-03-31 00:18:23,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully updated templates OVFs in Data Center Lok >2014-03-31 00:18:23,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 00:18:23,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 00:23:44,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 758572aa >2014-03-31 00:23:44,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: c2b332d >2014-03-31 00:23:44,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: c2b332d >2014-03-31 00:23:44,457 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 758572aa >2014-03-31 00:23:44,461 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 01:18:22,634 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-57) Start refreshing all users data >2014-03-31 01:18:23,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-03-31 01:18:23,003 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VM OVFs in Data Center Lok >2014-03-31 01:18:23,004 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated VM OVFs in Data Center Lok >2014-03-31 01:18:23,004 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update template OVFs in Data Center Lok >2014-03-31 01:18:23,004 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated templates OVFs in Data Center Lok >2014-03-31 01:18:23,004 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 01:18:23,005 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 01:23:44,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 26178d35 >2014-03-31 01:23:44,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5437a61c >2014-03-31 01:23:44,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5437a61c >2014-03-31 01:23:44,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 26178d35 >2014-03-31 01:23:44,714 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 02:18:22,636 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-31 02:18:23,006 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-03-31 02:18:23,007 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-03-31 02:18:23,007 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-03-31 02:18:23,007 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-03-31 02:18:23,008 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-03-31 02:18:23,008 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 02:18:23,008 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 02:23:44,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6484ec8a >2014-03-31 02:23:44,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 52c58b94 >2014-03-31 02:23:44,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 52c58b94 >2014-03-31 02:23:44,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6484ec8a >2014-03-31 02:23:44,948 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 03:18:22,637 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-03-31 03:18:23,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-03-31 03:18:23,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-03-31 03:18:23,012 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-03-31 03:18:23,012 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-03-31 03:18:23,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-03-31 03:18:23,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 03:18:23,014 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 03:23:44,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7adc102b >2014-03-31 03:23:44,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5a9b611 >2014-03-31 03:23:45,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a9b611 >2014-03-31 03:23:45,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7adc102b >2014-03-31 03:23:45,188 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-66) Start deleteAgedOutAuditLogs >2014-03-31 03:35:35,005 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-66) Finished deleteAgedOutAuditLogs >2014-03-31 04:18:22,639 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-03-31 04:18:23,014 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-03-31 04:18:23,015 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VM OVFs in Data Center Lok >2014-03-31 04:18:23,016 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated VM OVFs in Data Center Lok >2014-03-31 04:18:23,016 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update template OVFs in Data Center Lok >2014-03-31 04:18:23,017 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated templates OVFs in Data Center Lok >2014-03-31 04:18:23,017 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 04:18:23,018 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 04:23:45,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3de450b6 >2014-03-31 04:23:45,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 586efc78 >2014-03-31 04:23:45,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 586efc78 >2014-03-31 04:23:45,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3de450b6 >2014-03-31 04:23:45,430 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 05:18:22,640 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-76) Start refreshing all users data >2014-03-31 05:18:23,018 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-03-31 05:18:23,019 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VM OVFs in Data Center Lok >2014-03-31 05:18:23,020 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully updated VM OVFs in Data Center Lok >2014-03-31 05:18:23,020 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update template OVFs in Data Center Lok >2014-03-31 05:18:23,021 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully updated templates OVFs in Data Center Lok >2014-03-31 05:18:23,021 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 05:18:23,021 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 05:23:45,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5a4a3ff6 >2014-03-31 05:23:45,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 40ad5dc9 >2014-03-31 05:23:45,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 40ad5dc9 >2014-03-31 05:23:45,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a4a3ff6 >2014-03-31 05:23:45,671 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 06:18:22,642 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-76) Start refreshing all users data >2014-03-31 06:18:23,021 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-03-31 06:18:23,022 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VM OVFs in Data Center Lok >2014-03-31 06:18:23,023 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully updated VM OVFs in Data Center Lok >2014-03-31 06:18:23,023 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update template OVFs in Data Center Lok >2014-03-31 06:18:23,024 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully updated templates OVFs in Data Center Lok >2014-03-31 06:18:23,024 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 06:18:23,024 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 06:23:45,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 61dba7f4 >2014-03-31 06:23:45,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1befb65d >2014-03-31 06:23:45,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1befb65d >2014-03-31 06:23:45,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 61dba7f4 >2014-03-31 06:23:45,910 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 07:18:22,644 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-03-31 07:18:23,024 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-03-31 07:18:23,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-03-31 07:18:23,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-03-31 07:18:23,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-03-31 07:18:23,026 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-03-31 07:18:23,026 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 07:18:23,027 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 07:23:45,914 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4db9bc5d >2014-03-31 07:23:45,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5cfda588 >2014-03-31 07:23:46,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5cfda588 >2014-03-31 07:23:46,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4db9bc5d >2014-03-31 07:23:46,150 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 08:18:22,646 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-31 08:18:23,027 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-03-31 08:18:23,028 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VM OVFs in Data Center Lok >2014-03-31 08:18:23,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated VM OVFs in Data Center Lok >2014-03-31 08:18:23,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update template OVFs in Data Center Lok >2014-03-31 08:18:23,030 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated templates OVFs in Data Center Lok >2014-03-31 08:18:23,030 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 08:18:23,031 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 08:23:46,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1e311545 >2014-03-31 08:23:46,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 286b747f >2014-03-31 08:23:46,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 286b747f >2014-03-31 08:23:46,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1e311545 >2014-03-31 08:23:46,576 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 09:18:22,647 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-03-31 09:18:23,031 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-03-31 09:18:23,032 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VM OVFs in Data Center Lok >2014-03-31 09:18:23,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Successfully updated VM OVFs in Data Center Lok >2014-03-31 09:18:23,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update template OVFs in Data Center Lok >2014-03-31 09:18:23,034 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Successfully updated templates OVFs in Data Center Lok >2014-03-31 09:18:23,034 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 09:18:23,034 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 09:23:46,580 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 74cc693 >2014-03-31 09:23:46,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4180e0a3 >2014-03-31 09:23:46,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4180e0a3 >2014-03-31 09:23:46,812 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 74cc693 >2014-03-31 09:23:46,817 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 10:18:22,649 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-31 10:18:23,034 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-03-31 10:18:23,035 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VM OVFs in Data Center Lok >2014-03-31 10:18:23,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated VM OVFs in Data Center Lok >2014-03-31 10:18:23,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update template OVFs in Data Center Lok >2014-03-31 10:18:23,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated templates OVFs in Data Center Lok >2014-03-31 10:18:23,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 10:18:23,037 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 10:23:46,821 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1fb752a2 >2014-03-31 10:23:46,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3c9219f2 >2014-03-31 10:23:47,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c9219f2 >2014-03-31 10:23:47,159 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1fb752a2 >2014-03-31 10:23:47,164 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 11:18:22,653 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-31 11:18:23,037 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-03-31 11:18:23,038 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VM OVFs in Data Center Lok >2014-03-31 11:18:23,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated VM OVFs in Data Center Lok >2014-03-31 11:18:23,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update template OVFs in Data Center Lok >2014-03-31 11:18:23,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated templates OVFs in Data Center Lok >2014-03-31 11:18:23,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 11:18:23,040 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 11:23:47,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 76f128de >2014-03-31 11:23:47,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 52b243d3 >2014-03-31 11:23:47,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 52b243d3 >2014-03-31 11:23:47,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76f128de >2014-03-31 11:23:47,408 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 12:18:22,655 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-03-31 12:18:23,040 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-03-31 12:18:23,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-03-31 12:18:23,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-03-31 12:18:23,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-03-31 12:18:23,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-03-31 12:18:23,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 12:18:23,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 12:23:47,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4b860a94 >2014-03-31 12:23:47,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4d20a148 >2014-03-31 12:23:47,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4d20a148 >2014-03-31 12:23:47,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b860a94 >2014-03-31 12:23:47,646 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 13:18:22,657 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-03-31 13:18:23,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-03-31 13:18:23,043 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VM OVFs in Data Center Lok >2014-03-31 13:18:23,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated VM OVFs in Data Center Lok >2014-03-31 13:18:23,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update template OVFs in Data Center Lok >2014-03-31 13:18:23,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated templates OVFs in Data Center Lok >2014-03-31 13:18:23,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 13:18:23,045 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 13:23:47,650 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 463956b2 >2014-03-31 13:23:47,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: ae9004d >2014-03-31 13:23:48,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ae9004d >2014-03-31 13:23:48,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 463956b2 >2014-03-31 13:23:48,016 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 14:18:22,659 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-03-31 14:18:23,046 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-03-31 14:18:23,046 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VM OVFs in Data Center Lok >2014-03-31 14:18:23,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Successfully updated VM OVFs in Data Center Lok >2014-03-31 14:18:23,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update template OVFs in Data Center Lok >2014-03-31 14:18:23,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Successfully updated templates OVFs in Data Center Lok >2014-03-31 14:18:23,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 14:18:23,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 14:23:48,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5cd225f7 >2014-03-31 14:23:48,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 68baaabc >2014-03-31 14:23:48,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 68baaabc >2014-03-31 14:23:48,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5cd225f7 >2014-03-31 14:23:48,261 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 15:18:22,661 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-03-31 15:18:23,049 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-03-31 15:18:23,050 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VM OVFs in Data Center Lok >2014-03-31 15:18:23,051 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated VM OVFs in Data Center Lok >2014-03-31 15:18:23,051 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update template OVFs in Data Center Lok >2014-03-31 15:18:23,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated templates OVFs in Data Center Lok >2014-03-31 15:18:23,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 15:18:23,053 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 15:23:48,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5c7ee536 >2014-03-31 15:23:48,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: efa6bab >2014-03-31 15:23:48,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: efa6bab >2014-03-31 15:23:48,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5c7ee536 >2014-03-31 15:23:48,514 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 16:18:22,662 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-03-31 16:18:23,054 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-03-31 16:18:23,055 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VM OVFs in Data Center Lok >2014-03-31 16:18:23,055 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated VM OVFs in Data Center Lok >2014-03-31 16:18:23,055 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update template OVFs in Data Center Lok >2014-03-31 16:18:23,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated templates OVFs in Data Center Lok >2014-03-31 16:18:23,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 16:18:23,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 16:23:48,518 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2bc3ea37 >2014-03-31 16:23:48,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4b45d753 >2014-03-31 16:23:48,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b45d753 >2014-03-31 16:23:48,749 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2bc3ea37 >2014-03-31 16:23:48,754 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 17:18:22,664 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-03-31 17:18:23,057 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-03-31 17:18:23,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VM OVFs in Data Center Lok >2014-03-31 17:18:23,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated VM OVFs in Data Center Lok >2014-03-31 17:18:23,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update template OVFs in Data Center Lok >2014-03-31 17:18:23,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated templates OVFs in Data Center Lok >2014-03-31 17:18:23,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 17:18:23,060 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 17:23:48,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2d3ebdf >2014-03-31 17:23:48,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 27227eec >2014-03-31 17:23:49,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 27227eec >2014-03-31 17:23:49,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d3ebdf >2014-03-31 17:23:49,007 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 18:18:22,665 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-03-31 18:18:23,060 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-03-31 18:18:23,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VM OVFs in Data Center Lok >2014-03-31 18:18:23,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated VM OVFs in Data Center Lok >2014-03-31 18:18:23,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update template OVFs in Data Center Lok >2014-03-31 18:18:23,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated templates OVFs in Data Center Lok >2014-03-31 18:18:23,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 18:18:23,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 18:23:49,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 43fc3fc >2014-03-31 18:23:49,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 765bd791 >2014-03-31 18:23:49,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 765bd791 >2014-03-31 18:23:49,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 43fc3fc >2014-03-31 18:23:49,243 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 19:18:22,667 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-67) Start refreshing all users data >2014-03-31 19:18:23,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-03-31 19:18:23,064 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VM OVFs in Data Center Lok >2014-03-31 19:18:23,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated VM OVFs in Data Center Lok >2014-03-31 19:18:23,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update template OVFs in Data Center Lok >2014-03-31 19:18:23,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated templates OVFs in Data Center Lok >2014-03-31 19:18:23,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 19:18:23,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 19:23:49,246 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4d3238ce >2014-03-31 19:23:49,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1c2f83fa >2014-03-31 19:23:49,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1c2f83fa >2014-03-31 19:23:49,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4d3238ce >2014-03-31 19:23:49,481 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 20:18:22,668 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-03-31 20:18:23,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-03-31 20:18:23,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-03-31 20:18:23,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-03-31 20:18:23,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-03-31 20:18:23,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-03-31 20:18:23,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 20:18:23,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 20:23:49,485 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 459900e2 >2014-03-31 20:23:49,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 40091920 >2014-03-31 20:23:49,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 40091920 >2014-03-31 20:23:49,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 459900e2 >2014-03-31 20:23:49,722 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 21:18:22,670 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-03-31 21:18:23,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-03-31 21:18:23,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VM OVFs in Data Center Lok >2014-03-31 21:18:23,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated VM OVFs in Data Center Lok >2014-03-31 21:18:23,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update template OVFs in Data Center Lok >2014-03-31 21:18:23,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated templates OVFs in Data Center Lok >2014-03-31 21:18:23,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 21:18:23,071 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 21:23:49,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 50eed409 >2014-03-31 21:23:49,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6fa2c461 >2014-03-31 21:23:49,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6fa2c461 >2014-03-31 21:23:49,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 50eed409 >2014-03-31 21:23:49,962 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 22:18:22,670 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-03-31 22:18:23,071 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-03-31 22:18:23,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-03-31 22:18:23,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-03-31 22:18:23,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-03-31 22:18:23,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-03-31 22:18:23,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 22:18:23,074 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 22:23:49,967 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 17884e79 >2014-03-31 22:23:49,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 75d7f600 >2014-03-31 22:23:50,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 75d7f600 >2014-03-31 22:23:50,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 17884e79 >2014-03-31 22:23:50,203 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-03-31 23:18:22,672 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-67) Start refreshing all users data >2014-03-31 23:18:23,074 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-03-31 23:18:23,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VM OVFs in Data Center Lok >2014-03-31 23:18:23,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated VM OVFs in Data Center Lok >2014-03-31 23:18:23,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update template OVFs in Data Center Lok >2014-03-31 23:18:23,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated templates OVFs in Data Center Lok >2014-03-31 23:18:23,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-03-31 23:18:23,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-03-31 23:23:50,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 53ee8acb >2014-03-31 23:23:50,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 51da4079 >2014-03-31 23:23:50,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 51da4079 >2014-03-31 23:23:50,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 53ee8acb >2014-03-31 23:23:50,442 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 00:18:22,671 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-04-01 00:18:23,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-04-01 00:18:23,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VM OVFs in Data Center Lok >2014-04-01 00:18:23,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated VM OVFs in Data Center Lok >2014-04-01 00:18:23,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update template OVFs in Data Center Lok >2014-04-01 00:18:23,079 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated templates OVFs in Data Center Lok >2014-04-01 00:18:23,079 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 00:18:23,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 00:23:50,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 36471750 >2014-04-01 00:23:50,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 754ba106 >2014-04-01 00:23:50,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 754ba106 >2014-04-01 00:23:50,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 36471750 >2014-04-01 00:23:50,680 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 01:18:22,673 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-04-01 01:18:23,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-04-01 01:18:23,081 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VM OVFs in Data Center Lok >2014-04-01 01:18:23,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully updated VM OVFs in Data Center Lok >2014-04-01 01:18:23,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update template OVFs in Data Center Lok >2014-04-01 01:18:23,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully updated templates OVFs in Data Center Lok >2014-04-01 01:18:23,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 01:18:23,083 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 01:23:50,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 115db9f6 >2014-04-01 01:23:50,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 205bce62 >2014-04-01 01:23:50,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 205bce62 >2014-04-01 01:23:50,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 115db9f6 >2014-04-01 01:23:50,917 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 02:18:22,675 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-04-01 02:18:23,083 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-04-01 02:18:23,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VM OVFs in Data Center Lok >2014-04-01 02:18:23,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated VM OVFs in Data Center Lok >2014-04-01 02:18:23,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update template OVFs in Data Center Lok >2014-04-01 02:18:23,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated templates OVFs in Data Center Lok >2014-04-01 02:18:23,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 02:18:23,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 02:23:50,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1fc123c2 >2014-04-01 02:23:50,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4112a5d7 >2014-04-01 02:23:51,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4112a5d7 >2014-04-01 02:23:51,154 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1fc123c2 >2014-04-01 02:23:51,159 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 03:18:22,677 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-01 03:18:23,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-04-01 03:18:23,087 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VM OVFs in Data Center Lok >2014-04-01 03:18:23,087 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated VM OVFs in Data Center Lok >2014-04-01 03:18:23,087 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update template OVFs in Data Center Lok >2014-04-01 03:18:23,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated templates OVFs in Data Center Lok >2014-04-01 03:18:23,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 03:18:23,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 03:23:51,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 19f49c13 >2014-04-01 03:23:51,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: a32b0b5 >2014-04-01 03:23:51,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: a32b0b5 >2014-04-01 03:23:51,392 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 19f49c13 >2014-04-01 03:23:51,396 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-34) Start deleteAgedOutAuditLogs >2014-04-01 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-34) Finished deleteAgedOutAuditLogs >2014-04-01 04:18:22,678 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-04-01 04:18:23,089 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VMs/Templates Ovf. >2014-04-01 04:18:23,089 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update VM OVFs in Data Center Lok >2014-04-01 04:18:23,090 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Successfully updated VM OVFs in Data Center Lok >2014-04-01 04:18:23,090 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to update template OVFs in Data Center Lok >2014-04-01 04:18:23,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Successfully updated templates OVFs in Data Center Lok >2014-04-01 04:18:23,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 04:18:23,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 04:23:51,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5ea8e1d4 >2014-04-01 04:23:51,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 218255d9 >2014-04-01 04:23:51,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 218255d9 >2014-04-01 04:23:51,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5ea8e1d4 >2014-04-01 04:23:51,639 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 05:18:22,680 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-01 05:18:23,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-01 05:18:23,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-01 05:18:23,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-01 05:18:23,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-01 05:18:23,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-01 05:18:23,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 05:18:23,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 05:23:51,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 275e64b >2014-04-01 05:23:51,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: a624730 >2014-04-01 05:23:51,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: a624730 >2014-04-01 05:23:51,871 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 275e64b >2014-04-01 05:23:51,876 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 06:18:22,682 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-04-01 06:18:23,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-04-01 06:18:23,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VM OVFs in Data Center Lok >2014-04-01 06:18:23,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated VM OVFs in Data Center Lok >2014-04-01 06:18:23,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update template OVFs in Data Center Lok >2014-04-01 06:18:23,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated templates OVFs in Data Center Lok >2014-04-01 06:18:23,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 06:18:23,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 06:23:51,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3cf598f >2014-04-01 06:23:51,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2200bd3d >2014-04-01 06:23:52,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2200bd3d >2014-04-01 06:23:52,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3cf598f >2014-04-01 06:23:52,115 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 07:18:22,685 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-90) Start refreshing all users data >2014-04-01 07:18:23,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-04-01 07:18:23,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VM OVFs in Data Center Lok >2014-04-01 07:18:23,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated VM OVFs in Data Center Lok >2014-04-01 07:18:23,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update template OVFs in Data Center Lok >2014-04-01 07:18:23,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated templates OVFs in Data Center Lok >2014-04-01 07:18:23,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 07:18:23,100 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 07:23:52,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 653cde6c >2014-04-01 07:23:52,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7f2eb860 >2014-04-01 07:23:52,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f2eb860 >2014-04-01 07:23:52,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 653cde6c >2014-04-01 07:23:52,353 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 08:18:22,687 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-04-01 08:18:23,100 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-04-01 08:18:23,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VM OVFs in Data Center Lok >2014-04-01 08:18:23,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated VM OVFs in Data Center Lok >2014-04-01 08:18:23,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update template OVFs in Data Center Lok >2014-04-01 08:18:23,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated templates OVFs in Data Center Lok >2014-04-01 08:18:23,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 08:18:23,103 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 08:23:52,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: d46c3c6 >2014-04-01 08:23:52,360 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 20df02f9 >2014-04-01 08:23:52,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 20df02f9 >2014-04-01 08:23:52,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: d46c3c6 >2014-04-01 08:23:52,961 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 09:18:22,689 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-04-01 09:18:23,103 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-01 09:18:23,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-01 09:18:23,105 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-01 09:18:23,105 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-01 09:18:23,105 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-01 09:18:23,105 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 09:18:23,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 09:23:52,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6427fed1 >2014-04-01 09:23:52,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7dca7eb5 >2014-04-01 09:23:53,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7dca7eb5 >2014-04-01 09:23:53,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6427fed1 >2014-04-01 09:23:53,201 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 10:18:22,691 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-04-01 10:18:23,107 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-04-01 10:18:23,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VM OVFs in Data Center Lok >2014-04-01 10:18:23,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated VM OVFs in Data Center Lok >2014-04-01 10:18:23,109 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update template OVFs in Data Center Lok >2014-04-01 10:18:23,109 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated templates OVFs in Data Center Lok >2014-04-01 10:18:23,109 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 10:18:23,110 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 10:23:53,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4d3f81bf >2014-04-01 10:23:53,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7f27f9f7 >2014-04-01 10:23:53,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f27f9f7 >2014-04-01 10:23:53,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4d3f81bf >2014-04-01 10:23:53,496 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 11:18:22,692 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-67) Start refreshing all users data >2014-04-01 11:18:23,110 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-04-01 11:18:23,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VM OVFs in Data Center Lok >2014-04-01 11:18:23,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated VM OVFs in Data Center Lok >2014-04-01 11:18:23,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update template OVFs in Data Center Lok >2014-04-01 11:18:23,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated templates OVFs in Data Center Lok >2014-04-01 11:18:23,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 11:18:23,113 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 11:23:53,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 476c5e8b >2014-04-01 11:23:53,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1bb31c4f >2014-04-01 11:23:53,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1bb31c4f >2014-04-01 11:23:53,830 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 476c5e8b >2014-04-01 11:23:53,834 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 12:18:22,694 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-04-01 12:18:23,113 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-04-01 12:18:23,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VM OVFs in Data Center Lok >2014-04-01 12:18:23,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated VM OVFs in Data Center Lok >2014-04-01 12:18:23,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update template OVFs in Data Center Lok >2014-04-01 12:18:23,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated templates OVFs in Data Center Lok >2014-04-01 12:18:23,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 12:18:23,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 12:23:53,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4b3a5337 >2014-04-01 12:23:53,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 622d6676 >2014-04-01 12:23:54,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 622d6676 >2014-04-01 12:23:54,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b3a5337 >2014-04-01 12:23:54,075 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 13:18:22,696 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-04-01 13:18:23,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-04-01 13:18:23,116 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-04-01 13:18:23,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-04-01 13:18:23,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-04-01 13:18:23,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-04-01 13:18:23,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 13:18:23,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 13:23:54,078 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 32f6a4d4 >2014-04-01 13:23:54,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5ebf746f >2014-04-01 13:23:54,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5ebf746f >2014-04-01 13:23:54,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 32f6a4d4 >2014-04-01 13:23:54,315 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 13:36:34,951 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginAdminUserCommand internal: false. >2014-04-01 13:36:34,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-01 13:36:35,031 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-04-01 13:36:35,038 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-01 13:37:05,210 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-11) [58ce00ec] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-01 13:37:05,300 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [58ce00ec] Correlation ID: 58ce00ec, Call Stack: null, Custom Event ID: -1, Message: Data Center adf with Type NFS, Compatibility Version 3.3 and Quota Type HARD_ENFORCEMENT was added by admin@internal >2014-04-01 13:37:13,835 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-8) [36c9f350] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: 887f8af2-ed05-47f2-84ea-bf9fe7a93625 Type: StoragePool >2014-04-01 13:37:13,938 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) [36c9f350] Correlation ID: 36c9f350, Call Stack: null, Custom Event ID: -1, Message: Host cluster asdf was added by admin@internal >2014-04-01 13:39:13,809 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (pool-4-thread-47) [4ef176b] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: 887f8af2-ed05-47f2-84ea-bf9fe7a93625 Type: StoragePool >2014-04-01 13:39:13,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-47) [4ef176b] Correlation ID: 4ef176b, Job ID: b1abd202-628f-49b9-b7b7-d22b4e75ecee, Call Stack: null, Custom Event ID: -1, Message: Data Center adf was removed by admin@internal >2014-04-01 13:39:21,048 INFO [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (pool-4-thread-47) [3627c09c] Running command: RemoveVdsGroupCommand internal: false. Entities affected : ID: 2ea00c0b-e699-47d0-8354-e6d126dd9a57 Type: VdsGroups >2014-04-01 13:39:21,071 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-47) [3627c09c] Correlation ID: 3627c09c, Call Stack: null, Custom Event ID: -1, Message: Host cluster asdf was removed by admin@internal >2014-04-01 14:18:22,698 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-04-01 14:18:23,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-04-01 14:18:23,119 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VM OVFs in Data Center Lok >2014-04-01 14:18:23,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated VM OVFs in Data Center Lok >2014-04-01 14:18:23,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update template OVFs in Data Center Lok >2014-04-01 14:18:23,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated templates OVFs in Data Center Lok >2014-04-01 14:18:23,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 14:18:23,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 14:23:54,319 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5b7e48b4 >2014-04-01 14:23:54,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3fd75059 >2014-04-01 14:23:54,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3fd75059 >2014-04-01 14:23:54,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5b7e48b4 >2014-04-01 14:23:54,551 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 15:18:22,699 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-04-01 15:18:23,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-04-01 15:18:23,122 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VM OVFs in Data Center Lok >2014-04-01 15:18:23,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated VM OVFs in Data Center Lok >2014-04-01 15:18:23,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update template OVFs in Data Center Lok >2014-04-01 15:18:23,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated templates OVFs in Data Center Lok >2014-04-01 15:18:23,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 15:18:23,124 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 15:23:54,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: c2cdcf5 >2014-04-01 15:23:54,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 555e5f11 >2014-04-01 15:23:54,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 555e5f11 >2014-04-01 15:23:54,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: c2cdcf5 >2014-04-01 15:23:54,809 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 16:18:22,702 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-66) Start refreshing all users data >2014-04-01 16:18:23,124 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-01 16:18:23,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-01 16:18:23,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-01 16:18:23,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-01 16:18:23,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-01 16:18:23,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 16:18:23,127 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 16:23:54,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 431d5ab5 >2014-04-01 16:23:54,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3c2e437f >2014-04-01 16:23:55,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c2e437f >2014-04-01 16:23:55,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 431d5ab5 >2014-04-01 16:23:55,223 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 17:18:22,704 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-42) Start refreshing all users data >2014-04-01 17:18:23,127 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-01 17:18:23,128 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-01 17:18:23,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-01 17:18:23,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-01 17:18:23,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-01 17:18:23,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 17:18:23,130 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 17:23:55,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 14cfb962 >2014-04-01 17:23:55,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5193db9a >2014-04-01 17:23:55,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5193db9a >2014-04-01 17:23:55,759 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 14cfb962 >2014-04-01 17:23:55,763 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 18:18:22,706 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-01 18:18:23,130 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-04-01 18:18:23,131 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VM OVFs in Data Center Lok >2014-04-01 18:18:23,131 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated VM OVFs in Data Center Lok >2014-04-01 18:18:23,132 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update template OVFs in Data Center Lok >2014-04-01 18:18:23,132 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated templates OVFs in Data Center Lok >2014-04-01 18:18:23,132 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 18:18:23,132 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 18:23:55,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1bfba955 >2014-04-01 18:23:55,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 272a19cd >2014-04-01 18:23:55,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 272a19cd >2014-04-01 18:23:55,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1bfba955 >2014-04-01 18:23:56,003 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 19:18:22,708 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-01 19:18:23,132 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-04-01 19:18:23,133 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VM OVFs in Data Center Lok >2014-04-01 19:18:23,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated VM OVFs in Data Center Lok >2014-04-01 19:18:23,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update template OVFs in Data Center Lok >2014-04-01 19:18:23,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated templates OVFs in Data Center Lok >2014-04-01 19:18:23,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 19:18:23,135 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 19:23:56,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 34e9009 >2014-04-01 19:23:56,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1ad24091 >2014-04-01 19:23:56,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1ad24091 >2014-04-01 19:23:56,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 34e9009 >2014-04-01 19:23:56,403 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 20:18:22,708 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-04-01 20:18:23,135 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-01 20:18:23,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-01 20:18:23,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-01 20:18:23,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-01 20:18:23,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-01 20:18:23,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 20:18:23,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 20:23:56,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4bf61e2 >2014-04-01 20:23:56,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 30b33edd >2014-04-01 20:23:56,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 30b33edd >2014-04-01 20:23:56,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4bf61e2 >2014-04-01 20:23:56,651 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 21:18:22,709 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-04-01 21:18:23,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-04-01 21:18:23,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Lok >2014-04-01 21:18:23,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Lok >2014-04-01 21:18:23,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Lok >2014-04-01 21:18:23,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated templates OVFs in Data Center Lok >2014-04-01 21:18:23,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 21:18:23,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 21:23:56,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 69415bb7 >2014-04-01 21:23:56,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 41ec815d >2014-04-01 21:23:56,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 41ec815d >2014-04-01 21:23:56,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 69415bb7 >2014-04-01 21:23:56,896 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 22:18:22,711 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-04-01 22:18:23,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-04-01 22:18:23,141 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-04-01 22:18:23,141 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-04-01 22:18:23,141 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-04-01 22:18:23,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-04-01 22:18:23,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 22:18:23,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 22:23:56,900 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d5201c7 >2014-04-01 22:23:56,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 72f2beda >2014-04-01 22:23:57,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 72f2beda >2014-04-01 22:23:57,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d5201c7 >2014-04-01 22:23:57,333 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-01 23:18:22,713 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-04-01 23:18:23,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-04-01 23:18:23,144 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-04-01 23:18:23,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-04-01 23:18:23,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-04-01 23:18:23,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-04-01 23:18:23,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-01 23:18:23,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-01 23:23:57,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 622e7a5d >2014-04-01 23:23:57,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5963a206 >2014-04-01 23:23:57,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5963a206 >2014-04-01 23:23:57,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 622e7a5d >2014-04-01 23:23:57,579 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 00:18:22,715 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-04-02 00:18:23,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-04-02 00:18:23,147 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VM OVFs in Data Center Lok >2014-04-02 00:18:23,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated VM OVFs in Data Center Lok >2014-04-02 00:18:23,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update template OVFs in Data Center Lok >2014-04-02 00:18:23,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated templates OVFs in Data Center Lok >2014-04-02 00:18:23,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 00:18:23,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 00:23:57,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6443911e >2014-04-02 00:23:57,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7be875d3 >2014-04-02 00:23:57,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7be875d3 >2014-04-02 00:23:57,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6443911e >2014-04-02 00:23:57,905 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 01:18:22,718 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-04-02 01:18:23,150 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-02 01:18:23,150 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-02 01:18:23,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-02 01:18:23,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-02 01:18:23,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-02 01:18:23,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 01:18:23,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 01:23:57,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 17e7bf3a >2014-04-02 01:23:57,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6fbb8518 >2014-04-02 01:23:58,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6fbb8518 >2014-04-02 01:23:58,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 17e7bf3a >2014-04-02 01:23:58,150 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 02:18:22,720 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-04-02 02:18:23,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-04-02 02:18:23,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VM OVFs in Data Center Lok >2014-04-02 02:18:23,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated VM OVFs in Data Center Lok >2014-04-02 02:18:23,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update template OVFs in Data Center Lok >2014-04-02 02:18:23,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated templates OVFs in Data Center Lok >2014-04-02 02:18:23,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 02:18:23,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 02:23:58,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7f0d600a >2014-04-02 02:23:58,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 471c71b0 >2014-04-02 02:23:58,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 471c71b0 >2014-04-02 02:23:58,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f0d600a >2014-04-02 02:23:58,393 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 03:18:22,722 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-04-02 03:18:23,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VMs/Templates Ovf. >2014-04-02 03:18:23,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update VM OVFs in Data Center Lok >2014-04-02 03:18:23,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Successfully updated VM OVFs in Data Center Lok >2014-04-02 03:18:23,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to update template OVFs in Data Center Lok >2014-04-02 03:18:23,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Successfully updated templates OVFs in Data Center Lok >2014-04-02 03:18:23,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 03:18:23,159 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 03:23:58,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2029bd2c >2014-04-02 03:23:58,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1442a433 >2014-04-02 03:23:58,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1442a433 >2014-04-02 03:23:58,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2029bd2c >2014-04-02 03:23:58,672 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-31) Start deleteAgedOutAuditLogs >2014-04-02 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-31) Finished deleteAgedOutAuditLogs >2014-04-02 04:18:22,725 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-04-02 04:18:23,160 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-04-02 04:18:23,161 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VM OVFs in Data Center Lok >2014-04-02 04:18:23,162 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated VM OVFs in Data Center Lok >2014-04-02 04:18:23,162 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update template OVFs in Data Center Lok >2014-04-02 04:18:23,162 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated templates OVFs in Data Center Lok >2014-04-02 04:18:23,162 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 04:18:23,163 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 04:23:58,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5d5103c3 >2014-04-02 04:23:58,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 71ce2711 >2014-04-02 04:23:59,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 71ce2711 >2014-04-02 04:23:59,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5d5103c3 >2014-04-02 04:23:59,093 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 05:18:22,726 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-04-02 05:18:23,163 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-04-02 05:18:23,164 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VM OVFs in Data Center Lok >2014-04-02 05:18:23,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated VM OVFs in Data Center Lok >2014-04-02 05:18:23,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update template OVFs in Data Center Lok >2014-04-02 05:18:23,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated templates OVFs in Data Center Lok >2014-04-02 05:18:23,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 05:18:23,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 05:23:59,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 13c07c3e >2014-04-02 05:23:59,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 421cd493 >2014-04-02 05:23:59,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 421cd493 >2014-04-02 05:23:59,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 13c07c3e >2014-04-02 05:23:59,334 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 06:18:22,728 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-02 06:18:23,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-04-02 06:18:23,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VM OVFs in Data Center Lok >2014-04-02 06:18:23,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated VM OVFs in Data Center Lok >2014-04-02 06:18:23,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update template OVFs in Data Center Lok >2014-04-02 06:18:23,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated templates OVFs in Data Center Lok >2014-04-02 06:18:23,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 06:18:23,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 06:23:59,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 54c9520d >2014-04-02 06:23:59,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6d29fef7 >2014-04-02 06:23:59,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6d29fef7 >2014-04-02 06:23:59,754 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 54c9520d >2014-04-02 06:23:59,759 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 07:18:22,730 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-69) Start refreshing all users data >2014-04-02 07:18:23,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-04-02 07:18:23,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VM OVFs in Data Center Lok >2014-04-02 07:18:23,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated VM OVFs in Data Center Lok >2014-04-02 07:18:23,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update template OVFs in Data Center Lok >2014-04-02 07:18:23,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated templates OVFs in Data Center Lok >2014-04-02 07:18:23,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 07:18:23,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 07:23:59,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1576a1e6 >2014-04-02 07:23:59,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1f7f0e0c >2014-04-02 07:23:59,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1f7f0e0c >2014-04-02 07:23:59,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1576a1e6 >2014-04-02 07:24:00,002 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 08:18:22,732 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-86) Start refreshing all users data >2014-04-02 08:18:23,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-02 08:18:23,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-02 08:18:23,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-02 08:18:23,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-02 08:18:23,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-02 08:18:23,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 08:18:23,175 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 08:24:00,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4f24a98f >2014-04-02 08:24:00,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6ff813e0 >2014-04-02 08:24:00,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ff813e0 >2014-04-02 08:24:00,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f24a98f >2014-04-02 08:24:00,253 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 09:18:22,734 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-04-02 09:18:23,175 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-02 09:18:23,176 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-02 09:18:23,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-02 09:18:23,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-02 09:18:23,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-02 09:18:23,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 09:18:23,178 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 09:24:00,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4bec1c4 >2014-04-02 09:24:00,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7a32d4b >2014-04-02 09:24:00,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7a32d4b >2014-04-02 09:24:00,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4bec1c4 >2014-04-02 09:24:00,495 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 10:18:22,736 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-02 10:18:23,178 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-04-02 10:18:23,179 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VM OVFs in Data Center Lok >2014-04-02 10:18:23,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated VM OVFs in Data Center Lok >2014-04-02 10:18:23,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update template OVFs in Data Center Lok >2014-04-02 10:18:23,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated templates OVFs in Data Center Lok >2014-04-02 10:18:23,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 10:18:23,181 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 10:24:00,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 685c26b8 >2014-04-02 10:24:00,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7581cebe >2014-04-02 10:24:01,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7581cebe >2014-04-02 10:24:01,096 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 685c26b8 >2014-04-02 10:24:01,100 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 11:18:22,737 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-04-02 11:18:23,181 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-04-02 11:18:23,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VM OVFs in Data Center Lok >2014-04-02 11:18:23,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Successfully updated VM OVFs in Data Center Lok >2014-04-02 11:18:23,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update template OVFs in Data Center Lok >2014-04-02 11:18:23,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Successfully updated templates OVFs in Data Center Lok >2014-04-02 11:18:23,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 11:18:23,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 11:24:01,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1eb9469e >2014-04-02 11:24:01,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 15cb3ed4 >2014-04-02 11:24:01,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 15cb3ed4 >2014-04-02 11:24:01,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1eb9469e >2014-04-02 11:24:01,344 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 12:18:22,739 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-62) Start refreshing all users data >2014-04-02 12:18:23,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-04-02 12:18:23,185 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-04-02 12:18:23,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-04-02 12:18:23,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-04-02 12:18:23,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-04-02 12:18:23,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 12:18:23,187 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 12:24:01,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1b42ef75 >2014-04-02 12:24:01,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4776f84e >2014-04-02 12:24:01,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4776f84e >2014-04-02 12:24:01,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b42ef75 >2014-04-02 12:24:01,589 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 13:18:22,740 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-04-02 13:18:23,187 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-04-02 13:18:23,188 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VM OVFs in Data Center Lok >2014-04-02 13:18:23,188 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated VM OVFs in Data Center Lok >2014-04-02 13:18:23,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update template OVFs in Data Center Lok >2014-04-02 13:18:23,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated templates OVFs in Data Center Lok >2014-04-02 13:18:23,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 13:18:23,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 13:24:01,593 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4746e070 >2014-04-02 13:24:01,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7e38fd9e >2014-04-02 13:24:02,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7e38fd9e >2014-04-02 13:24:02,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4746e070 >2014-04-02 13:24:02,053 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 14:18:22,741 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-81) Start refreshing all users data >2014-04-02 14:18:23,190 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-04-02 14:18:23,191 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VM OVFs in Data Center Lok >2014-04-02 14:18:23,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated VM OVFs in Data Center Lok >2014-04-02 14:18:23,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update template OVFs in Data Center Lok >2014-04-02 14:18:23,193 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated templates OVFs in Data Center Lok >2014-04-02 14:18:23,193 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 14:18:23,193 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 14:24:02,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 70400917 >2014-04-02 14:24:02,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7c14fe93 >2014-04-02 14:24:02,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c14fe93 >2014-04-02 14:24:02,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 70400917 >2014-04-02 14:24:02,626 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 15:18:22,742 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-04-02 15:18:23,193 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-02 15:18:23,194 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-02 15:18:23,195 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-02 15:18:23,195 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-02 15:18:23,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-02 15:18:23,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 15:18:23,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 15:24:02,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4459c3ad >2014-04-02 15:24:02,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 613db5e9 >2014-04-02 15:24:03,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 613db5e9 >2014-04-02 15:24:03,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4459c3ad >2014-04-02 15:24:03,184 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 16:18:22,744 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-04-02 16:18:23,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-04-02 16:18:23,197 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VM OVFs in Data Center Lok >2014-04-02 16:18:23,198 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated VM OVFs in Data Center Lok >2014-04-02 16:18:23,198 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update template OVFs in Data Center Lok >2014-04-02 16:18:23,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated templates OVFs in Data Center Lok >2014-04-02 16:18:23,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 16:18:23,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 16:24:03,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6e1899e7 >2014-04-02 16:24:03,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 11204529 >2014-04-02 16:24:03,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 11204529 >2014-04-02 16:24:03,621 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6e1899e7 >2014-04-02 16:24:03,626 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 17:18:22,746 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-63) Start refreshing all users data >2014-04-02 17:18:23,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-04-02 17:18:23,200 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VM OVFs in Data Center Lok >2014-04-02 17:18:23,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated VM OVFs in Data Center Lok >2014-04-02 17:18:23,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update template OVFs in Data Center Lok >2014-04-02 17:18:23,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated templates OVFs in Data Center Lok >2014-04-02 17:18:23,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 17:18:23,202 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 17:24:03,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4595a98d >2014-04-02 17:24:03,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 762a4086 >2014-04-02 17:24:04,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 762a4086 >2014-04-02 17:24:04,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4595a98d >2014-04-02 17:24:04,174 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 18:18:22,748 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-04-02 18:18:23,203 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-02 18:18:23,203 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-02 18:18:23,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-02 18:18:23,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-02 18:18:23,205 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-02 18:18:23,205 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 18:18:23,205 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 18:24:04,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 388b544d >2014-04-02 18:24:04,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 434ba83b >2014-04-02 18:24:04,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 434ba83b >2014-04-02 18:24:04,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 388b544d >2014-04-02 18:24:04,446 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 19:18:22,750 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-04-02 19:18:23,206 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VMs/Templates Ovf. >2014-04-02 19:18:23,206 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VM OVFs in Data Center Lok >2014-04-02 19:18:23,208 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Successfully updated VM OVFs in Data Center Lok >2014-04-02 19:18:23,208 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update template OVFs in Data Center Lok >2014-04-02 19:18:23,208 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Successfully updated templates OVFs in Data Center Lok >2014-04-02 19:18:23,208 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 19:18:23,209 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 19:24:04,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6fc2715 >2014-04-02 19:24:04,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1469d748 >2014-04-02 19:24:04,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1469d748 >2014-04-02 19:24:04,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6fc2715 >2014-04-02 19:24:04,688 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 20:18:22,752 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-50) Start refreshing all users data >2014-04-02 20:18:23,209 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-02 20:18:23,210 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-02 20:18:23,211 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-02 20:18:23,211 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-02 20:18:23,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-02 20:18:23,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 20:18:23,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 20:24:04,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 331e6ab1 >2014-04-02 20:24:04,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 11eea4af >2014-04-02 20:24:04,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 11eea4af >2014-04-02 20:24:04,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 331e6ab1 >2014-04-02 20:24:04,927 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 21:18:22,754 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-04-02 21:18:23,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-04-02 21:18:23,213 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VM OVFs in Data Center Lok >2014-04-02 21:18:23,214 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated VM OVFs in Data Center Lok >2014-04-02 21:18:23,214 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update template OVFs in Data Center Lok >2014-04-02 21:18:23,215 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated templates OVFs in Data Center Lok >2014-04-02 21:18:23,215 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 21:18:23,215 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 21:24:04,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3683d7ef >2014-04-02 21:24:04,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 9d5a7df >2014-04-02 21:24:05,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 9d5a7df >2014-04-02 21:24:05,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3683d7ef >2014-04-02 21:24:05,173 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 22:18:22,757 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-04-02 22:18:23,216 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-04-02 22:18:23,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VM OVFs in Data Center Lok >2014-04-02 22:18:23,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated VM OVFs in Data Center Lok >2014-04-02 22:18:23,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update template OVFs in Data Center Lok >2014-04-02 22:18:23,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated templates OVFs in Data Center Lok >2014-04-02 22:18:23,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 22:18:23,219 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 22:24:05,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5426eb77 >2014-04-02 22:24:05,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 472cfcca >2014-04-02 22:24:05,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 472cfcca >2014-04-02 22:24:05,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5426eb77 >2014-04-02 22:24:05,734 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-02 23:18:22,761 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-02 23:18:23,219 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-04-02 23:18:23,220 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VM OVFs in Data Center Lok >2014-04-02 23:18:23,221 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully updated VM OVFs in Data Center Lok >2014-04-02 23:18:23,221 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update template OVFs in Data Center Lok >2014-04-02 23:18:23,221 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully updated templates OVFs in Data Center Lok >2014-04-02 23:18:23,221 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-02 23:18:23,222 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-02 23:24:05,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 54d2981f >2014-04-02 23:24:05,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4e0146ba >2014-04-02 23:24:05,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4e0146ba >2014-04-02 23:24:05,979 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 54d2981f >2014-04-02 23:24:05,984 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 00:18:22,763 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-04-03 00:18:23,222 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-04-03 00:18:23,224 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VM OVFs in Data Center Lok >2014-04-03 00:18:23,225 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated VM OVFs in Data Center Lok >2014-04-03 00:18:23,225 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update template OVFs in Data Center Lok >2014-04-03 00:18:23,225 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated templates OVFs in Data Center Lok >2014-04-03 00:18:23,225 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 00:18:23,226 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 00:24:05,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3e06f6e8 >2014-04-03 00:24:05,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3b3c1928 >2014-04-03 00:24:06,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3b3c1928 >2014-04-03 00:24:06,227 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3e06f6e8 >2014-04-03 00:24:06,232 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 01:18:22,765 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-04-03 01:18:23,226 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-04-03 01:18:23,227 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VM OVFs in Data Center Lok >2014-04-03 01:18:23,228 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated VM OVFs in Data Center Lok >2014-04-03 01:18:23,228 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update template OVFs in Data Center Lok >2014-04-03 01:18:23,228 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated templates OVFs in Data Center Lok >2014-04-03 01:18:23,228 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 01:18:23,229 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 01:24:06,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: bd1812d >2014-04-03 01:24:06,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6777b76a >2014-04-03 01:24:06,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6777b76a >2014-04-03 01:24:06,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: bd1812d >2014-04-03 01:24:06,473 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 02:18:22,768 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-04-03 02:18:23,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-03 02:18:23,231 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-04-03 02:18:23,232 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-04-03 02:18:23,232 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-04-03 02:18:23,232 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-04-03 02:18:23,232 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 02:18:23,233 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 02:24:06,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 9aa4bad >2014-04-03 02:24:06,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: b4e4fd0 >2014-04-03 02:24:06,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: b4e4fd0 >2014-04-03 02:24:06,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 9aa4bad >2014-04-03 02:24:06,723 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 03:18:22,769 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-04-03 03:18:23,233 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-03 03:18:23,234 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-03 03:18:23,235 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-03 03:18:23,235 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-03 03:18:23,236 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-03 03:18:23,236 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 03:18:23,236 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 03:24:06,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 28fce1ed >2014-04-03 03:24:06,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7254a3a0 >2014-04-03 03:24:06,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7254a3a0 >2014-04-03 03:24:06,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 28fce1ed >2014-04-03 03:24:06,964 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-31) Start deleteAgedOutAuditLogs >2014-04-03 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-31) Finished deleteAgedOutAuditLogs >2014-04-03 04:18:22,771 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-57) Start refreshing all users data >2014-04-03 04:18:23,236 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-04-03 04:18:23,237 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VM OVFs in Data Center Lok >2014-04-03 04:18:23,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated VM OVFs in Data Center Lok >2014-04-03 04:18:23,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update template OVFs in Data Center Lok >2014-04-03 04:18:23,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated templates OVFs in Data Center Lok >2014-04-03 04:18:23,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 04:18:23,239 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 04:24:06,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3853ebf9 >2014-04-03 04:24:06,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: da94553 >2014-04-03 04:24:07,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: da94553 >2014-04-03 04:24:07,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3853ebf9 >2014-04-03 04:24:07,206 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 05:18:22,773 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-04-03 05:18:23,239 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-04-03 05:18:23,240 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VM OVFs in Data Center Lok >2014-04-03 05:18:23,241 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated VM OVFs in Data Center Lok >2014-04-03 05:18:23,241 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update template OVFs in Data Center Lok >2014-04-03 05:18:23,241 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated templates OVFs in Data Center Lok >2014-04-03 05:18:23,241 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 05:18:23,242 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 05:24:07,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: c5274a5 >2014-04-03 05:24:07,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 796802f4 >2014-04-03 05:24:07,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 796802f4 >2014-04-03 05:24:07,443 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: c5274a5 >2014-04-03 05:24:07,450 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 06:18:22,775 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-90) Start refreshing all users data >2014-04-03 06:18:23,242 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-03 06:18:23,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-03 06:18:23,244 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-03 06:18:23,244 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-03 06:18:23,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-03 06:18:23,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 06:18:23,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 06:24:07,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: d050fcb >2014-04-03 06:24:07,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 474fdf1a >2014-04-03 06:24:07,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 474fdf1a >2014-04-03 06:24:07,687 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: d050fcb >2014-04-03 06:24:07,691 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 07:18:22,777 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-90) Start refreshing all users data >2014-04-03 07:18:23,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-04-03 07:18:23,247 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VM OVFs in Data Center Lok >2014-04-03 07:18:23,247 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully updated VM OVFs in Data Center Lok >2014-04-03 07:18:23,247 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update template OVFs in Data Center Lok >2014-04-03 07:18:23,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully updated templates OVFs in Data Center Lok >2014-04-03 07:18:23,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 07:18:23,249 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 07:24:07,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6418ff66 >2014-04-03 07:24:07,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2527b4e1 >2014-04-03 07:24:07,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2527b4e1 >2014-04-03 07:24:07,934 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6418ff66 >2014-04-03 07:24:07,940 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 08:18:22,778 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-04-03 08:18:23,249 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-04-03 08:18:23,250 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VM OVFs in Data Center Lok >2014-04-03 08:18:23,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated VM OVFs in Data Center Lok >2014-04-03 08:18:23,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update template OVFs in Data Center Lok >2014-04-03 08:18:23,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated templates OVFs in Data Center Lok >2014-04-03 08:18:23,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 08:18:23,252 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 08:24:07,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 361fc83a >2014-04-03 08:24:07,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 60927398 >2014-04-03 08:24:08,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 60927398 >2014-04-03 08:24:08,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 361fc83a >2014-04-03 08:24:08,187 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 09:18:22,780 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-04-03 09:18:23,252 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-03 09:18:23,253 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-03 09:18:23,254 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-03 09:18:23,254 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-03 09:18:23,254 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-03 09:18:23,254 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 09:18:23,255 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 09:24:08,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 12009dfc >2014-04-03 09:24:08,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7dbbcb76 >2014-04-03 09:24:08,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7dbbcb76 >2014-04-03 09:24:08,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 12009dfc >2014-04-03 09:24:08,753 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 10:18:22,783 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-04-03 10:18:23,256 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-04-03 10:18:23,256 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VM OVFs in Data Center Lok >2014-04-03 10:18:23,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated VM OVFs in Data Center Lok >2014-04-03 10:18:23,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update template OVFs in Data Center Lok >2014-04-03 10:18:23,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated templates OVFs in Data Center Lok >2014-04-03 10:18:23,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 10:18:23,258 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 10:24:08,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 55396092 >2014-04-03 10:24:08,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 29f33cf4 >2014-04-03 10:24:09,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 29f33cf4 >2014-04-03 10:24:09,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 55396092 >2014-04-03 10:24:09,006 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 11:18:22,784 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-04-03 11:18:23,259 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-04-03 11:18:23,260 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VM OVFs in Data Center Lok >2014-04-03 11:18:23,260 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated VM OVFs in Data Center Lok >2014-04-03 11:18:23,260 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update template OVFs in Data Center Lok >2014-04-03 11:18:23,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated templates OVFs in Data Center Lok >2014-04-03 11:18:23,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 11:18:23,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 11:24:09,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: c161b46 >2014-04-03 11:24:09,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7f5896dc >2014-04-03 11:24:09,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f5896dc >2014-04-03 11:24:09,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: c161b46 >2014-04-03 11:24:09,249 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 12:18:22,786 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-03 12:18:23,263 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-04-03 12:18:23,264 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VM OVFs in Data Center Lok >2014-04-03 12:18:23,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated VM OVFs in Data Center Lok >2014-04-03 12:18:23,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update template OVFs in Data Center Lok >2014-04-03 12:18:23,266 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated templates OVFs in Data Center Lok >2014-04-03 12:18:23,266 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 12:18:23,266 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 12:24:09,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4050eb83 >2014-04-03 12:24:09,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 960672 >2014-04-03 12:24:09,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 960672 >2014-04-03 12:24:09,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4050eb83 >2014-04-03 12:24:09,530 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 13:18:22,788 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-04-03 13:18:23,267 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-04-03 13:18:23,268 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-04-03 13:18:23,269 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-04-03 13:18:23,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-04-03 13:18:23,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-04-03 13:18:23,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 13:18:23,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 13:24:09,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 188602af >2014-04-03 13:24:09,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 323f9cc1 >2014-04-03 13:24:10,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 323f9cc1 >2014-04-03 13:24:10,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 188602af >2014-04-03 13:24:10,115 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 14:18:22,789 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-92) Start refreshing all users data >2014-04-03 14:18:23,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-04-03 14:18:23,272 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VM OVFs in Data Center Lok >2014-04-03 14:18:23,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully updated VM OVFs in Data Center Lok >2014-04-03 14:18:23,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update template OVFs in Data Center Lok >2014-04-03 14:18:23,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully updated templates OVFs in Data Center Lok >2014-04-03 14:18:23,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 14:18:23,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 14:24:10,118 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 521210b2 >2014-04-03 14:24:10,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 20064036 >2014-04-03 14:24:10,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 20064036 >2014-04-03 14:24:10,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 521210b2 >2014-04-03 14:24:10,371 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 15:18:22,791 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-03 15:18:23,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-04-03 15:18:23,276 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VM OVFs in Data Center Lok >2014-04-03 15:18:23,278 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated VM OVFs in Data Center Lok >2014-04-03 15:18:23,278 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update template OVFs in Data Center Lok >2014-04-03 15:18:23,278 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated templates OVFs in Data Center Lok >2014-04-03 15:18:23,278 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 15:18:23,279 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 15:24:10,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5ffee125 >2014-04-03 15:24:10,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 566f755f >2014-04-03 15:24:10,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 566f755f >2014-04-03 15:24:10,663 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5ffee125 >2014-04-03 15:24:10,667 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 16:18:22,793 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-04-03 16:18:23,279 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-04-03 16:18:23,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VM OVFs in Data Center Lok >2014-04-03 16:18:23,281 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated VM OVFs in Data Center Lok >2014-04-03 16:18:23,281 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update template OVFs in Data Center Lok >2014-04-03 16:18:23,282 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated templates OVFs in Data Center Lok >2014-04-03 16:18:23,282 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 16:18:23,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 16:24:10,671 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 610127bd >2014-04-03 16:24:11,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4fab96e5 >2014-04-03 16:24:11,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4fab96e5 >2014-04-03 16:24:11,250 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 610127bd >2014-04-03 16:24:11,258 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 17:18:22,795 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-04-03 17:18:23,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-04-03 17:18:23,284 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VM OVFs in Data Center Lok >2014-04-03 17:18:23,285 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated VM OVFs in Data Center Lok >2014-04-03 17:18:23,285 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update template OVFs in Data Center Lok >2014-04-03 17:18:23,286 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated templates OVFs in Data Center Lok >2014-04-03 17:18:23,286 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 17:18:23,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 17:24:11,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7025b47c >2014-04-03 17:24:11,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3b6c7269 >2014-04-03 17:24:11,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3b6c7269 >2014-04-03 17:24:11,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7025b47c >2014-04-03 17:24:11,567 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 18:18:22,795 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-03 18:18:23,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-03 18:18:23,288 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-03 18:18:23,289 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-03 18:18:23,289 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-03 18:18:23,290 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-03 18:18:23,290 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 18:18:23,290 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 18:24:11,572 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 9e48759 >2014-04-03 18:24:11,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7a71e921 >2014-04-03 18:24:11,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7a71e921 >2014-04-03 18:24:11,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 9e48759 >2014-04-03 18:24:11,809 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 19:18:22,797 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-03 19:18:23,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-04-03 19:18:23,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VM OVFs in Data Center Lok >2014-04-03 19:18:23,292 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated VM OVFs in Data Center Lok >2014-04-03 19:18:23,292 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update template OVFs in Data Center Lok >2014-04-03 19:18:23,292 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated templates OVFs in Data Center Lok >2014-04-03 19:18:23,292 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 19:18:23,293 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 19:24:11,811 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5d5cec1e >2014-04-03 19:24:11,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7f91d574 >2014-04-03 19:24:12,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f91d574 >2014-04-03 19:24:12,047 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5d5cec1e >2014-04-03 19:24:12,052 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 20:18:22,797 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-04-03 20:18:23,293 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-04-03 20:18:23,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VM OVFs in Data Center Lok >2014-04-03 20:18:23,295 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated VM OVFs in Data Center Lok >2014-04-03 20:18:23,295 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update template OVFs in Data Center Lok >2014-04-03 20:18:23,295 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated templates OVFs in Data Center Lok >2014-04-03 20:18:23,295 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 20:18:23,296 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 20:24:12,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7bca9a1b >2014-04-03 20:24:12,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 324ba21b >2014-04-03 20:24:12,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 324ba21b >2014-04-03 20:24:12,507 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7bca9a1b >2014-04-03 20:24:12,511 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 21:18:22,799 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-04-03 21:18:23,296 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-04-03 21:18:23,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VM OVFs in Data Center Lok >2014-04-03 21:18:23,299 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated VM OVFs in Data Center Lok >2014-04-03 21:18:23,299 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update template OVFs in Data Center Lok >2014-04-03 21:18:23,300 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated templates OVFs in Data Center Lok >2014-04-03 21:18:23,300 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 21:18:23,301 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 21:24:12,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6eae6347 >2014-04-03 21:24:12,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7af80315 >2014-04-03 21:24:12,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7af80315 >2014-04-03 21:24:12,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6eae6347 >2014-04-03 21:24:12,761 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 22:18:22,801 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-03 22:18:23,301 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-04-03 22:18:23,302 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VM OVFs in Data Center Lok >2014-04-03 22:18:23,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully updated VM OVFs in Data Center Lok >2014-04-03 22:18:23,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update template OVFs in Data Center Lok >2014-04-03 22:18:23,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully updated templates OVFs in Data Center Lok >2014-04-03 22:18:23,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 22:18:23,304 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 22:24:12,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 11322b91 >2014-04-03 22:24:12,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 74ca4568 >2014-04-03 22:24:12,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 74ca4568 >2014-04-03 22:24:12,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 11322b91 >2014-04-03 22:24:13,002 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-03 23:18:22,803 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-04-03 23:18:23,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-04-03 23:18:23,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VM OVFs in Data Center Lok >2014-04-03 23:18:23,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated VM OVFs in Data Center Lok >2014-04-03 23:18:23,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update template OVFs in Data Center Lok >2014-04-03 23:18:23,307 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated templates OVFs in Data Center Lok >2014-04-03 23:18:23,307 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-03 23:18:23,308 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-03 23:24:13,005 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 731e2af0 >2014-04-03 23:24:13,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 41952d9b >2014-04-03 23:24:13,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 41952d9b >2014-04-03 23:24:13,418 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 731e2af0 >2014-04-03 23:24:13,422 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 00:18:22,804 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-04-04 00:18:23,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-04 00:18:23,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-04 00:18:23,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-04 00:18:23,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-04 00:18:23,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-04 00:18:23,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 00:18:23,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 00:24:13,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 552a7ce0 >2014-04-04 00:24:13,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 586c81ce >2014-04-04 00:24:13,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 586c81ce >2014-04-04 00:24:13,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 552a7ce0 >2014-04-04 00:24:13,665 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 01:18:22,806 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-04-04 01:18:23,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-04-04 01:18:23,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VM OVFs in Data Center Lok >2014-04-04 01:18:23,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated VM OVFs in Data Center Lok >2014-04-04 01:18:23,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update template OVFs in Data Center Lok >2014-04-04 01:18:23,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated templates OVFs in Data Center Lok >2014-04-04 01:18:23,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 01:18:23,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 01:24:13,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5c1315e6 >2014-04-04 01:24:13,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6902de43 >2014-04-04 01:24:13,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6902de43 >2014-04-04 01:24:13,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5c1315e6 >2014-04-04 01:24:13,911 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 02:18:22,806 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-04 02:18:23,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-04 02:18:23,314 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-04 02:18:23,315 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-04 02:18:23,315 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-04 02:18:23,315 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-04 02:18:23,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 02:18:23,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 02:24:13,915 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 44a4085a >2014-04-04 02:24:13,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6d206fea >2014-04-04 02:24:14,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6d206fea >2014-04-04 02:24:14,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 44a4085a >2014-04-04 02:24:14,167 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 03:18:22,809 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-04-04 03:18:23,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-04-04 03:18:23,317 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VM OVFs in Data Center Lok >2014-04-04 03:18:23,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated VM OVFs in Data Center Lok >2014-04-04 03:18:23,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update template OVFs in Data Center Lok >2014-04-04 03:18:23,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated templates OVFs in Data Center Lok >2014-04-04 03:18:23,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 03:18:23,319 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 03:24:14,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 28e613bd >2014-04-04 03:24:14,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1919f245 >2014-04-04 03:24:14,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1919f245 >2014-04-04 03:24:14,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 28e613bd >2014-04-04 03:24:14,411 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-2) Start deleteAgedOutAuditLogs >2014-04-04 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-2) Finished deleteAgedOutAuditLogs >2014-04-04 04:18:22,811 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-04-04 04:18:23,319 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-04-04 04:18:23,320 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VM OVFs in Data Center Lok >2014-04-04 04:18:23,321 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated VM OVFs in Data Center Lok >2014-04-04 04:18:23,321 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update template OVFs in Data Center Lok >2014-04-04 04:18:23,321 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated templates OVFs in Data Center Lok >2014-04-04 04:18:23,321 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 04:18:23,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 04:24:14,415 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3a3b6dc4 >2014-04-04 04:24:14,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 170039ee >2014-04-04 04:24:14,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 170039ee >2014-04-04 04:24:14,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3a3b6dc4 >2014-04-04 04:24:14,710 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 05:18:22,813 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-04-04 05:18:23,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-04-04 05:18:23,323 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VM OVFs in Data Center Lok >2014-04-04 05:18:23,324 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated VM OVFs in Data Center Lok >2014-04-04 05:18:23,324 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update template OVFs in Data Center Lok >2014-04-04 05:18:23,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated templates OVFs in Data Center Lok >2014-04-04 05:18:23,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 05:18:23,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 05:24:14,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 34eb579a >2014-04-04 05:24:14,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 46c45a9a >2014-04-04 05:24:14,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 46c45a9a >2014-04-04 05:24:14,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 34eb579a >2014-04-04 05:24:14,953 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 06:18:22,815 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-86) Start refreshing all users data >2014-04-04 06:18:23,326 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-04-04 06:18:23,327 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VM OVFs in Data Center Lok >2014-04-04 06:18:23,327 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated VM OVFs in Data Center Lok >2014-04-04 06:18:23,327 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update template OVFs in Data Center Lok >2014-04-04 06:18:23,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated templates OVFs in Data Center Lok >2014-04-04 06:18:23,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 06:18:23,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 06:24:14,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6ce7ead5 >2014-04-04 06:24:14,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6dea7dce >2014-04-04 06:24:15,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6dea7dce >2014-04-04 06:24:15,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ce7ead5 >2014-04-04 06:24:15,200 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 07:18:22,817 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-04-04 07:18:23,329 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-04-04 07:18:23,331 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VM OVFs in Data Center Lok >2014-04-04 07:18:23,332 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated VM OVFs in Data Center Lok >2014-04-04 07:18:23,332 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update template OVFs in Data Center Lok >2014-04-04 07:18:23,333 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated templates OVFs in Data Center Lok >2014-04-04 07:18:23,333 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 07:18:23,334 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 07:24:15,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 15e3fa6b >2014-04-04 07:24:15,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2a4d9207 >2014-04-04 07:24:15,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2a4d9207 >2014-04-04 07:24:15,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 15e3fa6b >2014-04-04 07:24:15,447 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 08:18:22,819 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-04-04 08:18:23,334 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-04-04 08:18:23,335 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VM OVFs in Data Center Lok >2014-04-04 08:18:23,336 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully updated VM OVFs in Data Center Lok >2014-04-04 08:18:23,336 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update template OVFs in Data Center Lok >2014-04-04 08:18:23,337 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully updated templates OVFs in Data Center Lok >2014-04-04 08:18:23,337 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 08:18:23,337 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 08:24:15,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 53708c5d >2014-04-04 08:24:15,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 20f5877 >2014-04-04 08:24:15,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 20f5877 >2014-04-04 08:24:15,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 53708c5d >2014-04-04 08:24:15,694 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 09:18:22,821 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-04-04 09:18:23,337 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-04-04 09:18:23,338 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VM OVFs in Data Center Lok >2014-04-04 09:18:23,339 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated VM OVFs in Data Center Lok >2014-04-04 09:18:23,339 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update template OVFs in Data Center Lok >2014-04-04 09:18:23,339 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated templates OVFs in Data Center Lok >2014-04-04 09:18:23,339 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 09:18:23,340 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 09:24:15,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 31ab5bae >2014-04-04 09:24:15,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 691015fb >2014-04-04 09:24:15,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 691015fb >2014-04-04 09:24:15,952 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 31ab5bae >2014-04-04 09:24:15,974 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 10:18:22,823 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-04-04 10:18:23,340 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-04-04 10:18:23,341 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-04-04 10:18:23,342 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-04-04 10:18:23,342 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-04-04 10:18:23,343 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-04-04 10:18:23,343 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 10:18:23,344 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 10:24:15,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6ab11cbb >2014-04-04 10:24:15,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 11d56fb6 >2014-04-04 10:24:16,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 11d56fb6 >2014-04-04 10:24:16,450 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ab11cbb >2014-04-04 10:24:16,456 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 11:18:22,824 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-04-04 11:18:23,344 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-04-04 11:18:23,345 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VM OVFs in Data Center Lok >2014-04-04 11:18:23,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated VM OVFs in Data Center Lok >2014-04-04 11:18:23,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update template OVFs in Data Center Lok >2014-04-04 11:18:23,347 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated templates OVFs in Data Center Lok >2014-04-04 11:18:23,347 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 11:18:23,347 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 11:24:16,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2781f03b >2014-04-04 11:24:16,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 43c2c5c0 >2014-04-04 11:24:16,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 43c2c5c0 >2014-04-04 11:24:16,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2781f03b >2014-04-04 11:24:16,695 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 12:18:22,825 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-04 12:18:23,347 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-04 12:18:23,348 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-04 12:18:23,349 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-04 12:18:23,349 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-04 12:18:23,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-04 12:18:23,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 12:18:23,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 12:24:16,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 34430e8e >2014-04-04 12:24:16,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1e70a770 >2014-04-04 12:24:17,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1e70a770 >2014-04-04 12:24:17,148 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 34430e8e >2014-04-04 12:24:17,152 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 13:18:22,827 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-04-04 13:18:23,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-04-04 13:18:23,351 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VM OVFs in Data Center Lok >2014-04-04 13:18:23,352 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated VM OVFs in Data Center Lok >2014-04-04 13:18:23,352 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update template OVFs in Data Center Lok >2014-04-04 13:18:23,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated templates OVFs in Data Center Lok >2014-04-04 13:18:23,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 13:18:23,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 13:24:17,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 631fd477 >2014-04-04 13:24:17,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3eb60378 >2014-04-04 13:24:17,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3eb60378 >2014-04-04 13:24:17,394 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 631fd477 >2014-04-04 13:24:17,399 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 14:18:22,828 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-04 14:18:23,354 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-04-04 14:18:23,354 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VM OVFs in Data Center Lok >2014-04-04 14:18:23,355 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated VM OVFs in Data Center Lok >2014-04-04 14:18:23,355 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update template OVFs in Data Center Lok >2014-04-04 14:18:23,356 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated templates OVFs in Data Center Lok >2014-04-04 14:18:23,356 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 14:18:23,357 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 14:24:17,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4b7ce464 >2014-04-04 14:24:17,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 747687f3 >2014-04-04 14:24:22,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 747687f3 >2014-04-04 14:24:22,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b7ce464 >2014-04-04 14:24:22,850 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 15:18:22,830 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-42) Start refreshing all users data >2014-04-04 15:18:23,357 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-04-04 15:18:23,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VM OVFs in Data Center Lok >2014-04-04 15:18:23,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated VM OVFs in Data Center Lok >2014-04-04 15:18:23,359 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update template OVFs in Data Center Lok >2014-04-04 15:18:23,359 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated templates OVFs in Data Center Lok >2014-04-04 15:18:23,359 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 15:18:23,360 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 15:24:22,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3adf254f >2014-04-04 15:24:22,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 100a83bb >2014-04-04 15:24:23,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 100a83bb >2014-04-04 15:24:23,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3adf254f >2014-04-04 15:24:23,120 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 16:18:22,832 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-04-04 16:18:23,360 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-04-04 16:18:23,361 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-04-04 16:18:23,362 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-04-04 16:18:23,362 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-04-04 16:18:23,363 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-04-04 16:18:23,363 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 16:18:23,363 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 16:24:23,123 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 459ff071 >2014-04-04 16:24:23,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 762ae6d >2014-04-04 16:24:23,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 762ae6d >2014-04-04 16:24:23,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 459ff071 >2014-04-04 16:24:23,519 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 17:18:22,833 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-04-04 17:18:23,363 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-04-04 17:18:23,364 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-04-04 17:18:23,364 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-04-04 17:18:23,364 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-04-04 17:18:23,365 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-04-04 17:18:23,365 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 17:18:23,365 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 17:24:23,522 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4b3d1507 >2014-04-04 17:24:23,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 172fa528 >2014-04-04 17:24:23,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 172fa528 >2014-04-04 17:24:23,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b3d1507 >2014-04-04 17:24:23,763 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 18:18:22,834 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-04-04 18:18:23,365 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-04-04 18:18:23,366 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VM OVFs in Data Center Lok >2014-04-04 18:18:23,367 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated VM OVFs in Data Center Lok >2014-04-04 18:18:23,367 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update template OVFs in Data Center Lok >2014-04-04 18:18:23,367 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated templates OVFs in Data Center Lok >2014-04-04 18:18:23,367 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 18:18:23,368 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 18:24:23,768 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2b80f625 >2014-04-04 18:24:23,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4f511f23 >2014-04-04 18:24:24,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f511f23 >2014-04-04 18:24:24,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2b80f625 >2014-04-04 18:24:24,206 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 19:18:22,836 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-41) Start refreshing all users data >2014-04-04 19:18:23,368 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-04-04 19:18:23,369 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VM OVFs in Data Center Lok >2014-04-04 19:18:23,370 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated VM OVFs in Data Center Lok >2014-04-04 19:18:23,370 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update template OVFs in Data Center Lok >2014-04-04 19:18:23,370 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated templates OVFs in Data Center Lok >2014-04-04 19:18:23,370 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 19:18:23,371 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 19:24:24,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7f982835 >2014-04-04 19:24:24,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5e77f104 >2014-04-04 19:24:24,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e77f104 >2014-04-04 19:24:24,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f982835 >2014-04-04 19:24:24,452 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 20:18:22,836 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-04-04 20:18:23,372 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-04 20:18:23,372 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-04 20:18:23,373 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-04 20:18:23,373 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-04 20:18:23,373 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-04 20:18:23,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 20:18:23,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 20:24:24,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: daef48e >2014-04-04 20:24:24,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 59399da6 >2014-04-04 20:24:24,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 59399da6 >2014-04-04 20:24:24,725 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: daef48e >2014-04-04 20:24:24,729 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 21:18:22,838 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-67) Start refreshing all users data >2014-04-04 21:18:23,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-04-04 21:18:23,375 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VM OVFs in Data Center Lok >2014-04-04 21:18:23,376 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated VM OVFs in Data Center Lok >2014-04-04 21:18:23,376 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update template OVFs in Data Center Lok >2014-04-04 21:18:23,376 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated templates OVFs in Data Center Lok >2014-04-04 21:18:23,376 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 21:18:23,377 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 21:24:24,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6dba7266 >2014-04-04 21:24:24,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6bacbed1 >2014-04-04 21:24:24,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6bacbed1 >2014-04-04 21:24:24,968 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6dba7266 >2014-04-04 21:24:24,973 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 22:18:22,840 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-04-04 22:18:23,377 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-04-04 22:18:23,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VM OVFs in Data Center Lok >2014-04-04 22:18:23,380 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated VM OVFs in Data Center Lok >2014-04-04 22:18:23,380 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update template OVFs in Data Center Lok >2014-04-04 22:18:23,380 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated templates OVFs in Data Center Lok >2014-04-04 22:18:23,380 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 22:18:23,381 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 22:24:24,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 41e2155b >2014-04-04 22:24:24,979 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 61e82b27 >2014-04-04 22:24:25,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 61e82b27 >2014-04-04 22:24:25,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 41e2155b >2014-04-04 22:24:25,221 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-04 23:18:22,842 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-04-04 23:18:23,381 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-04-04 23:18:23,382 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VM OVFs in Data Center Lok >2014-04-04 23:18:23,383 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully updated VM OVFs in Data Center Lok >2014-04-04 23:18:23,383 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update template OVFs in Data Center Lok >2014-04-04 23:18:23,384 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully updated templates OVFs in Data Center Lok >2014-04-04 23:18:23,384 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-04 23:18:23,384 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-04 23:24:25,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 25c609e6 >2014-04-04 23:24:25,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6cb3ee69 >2014-04-04 23:24:25,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6cb3ee69 >2014-04-04 23:24:25,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 25c609e6 >2014-04-04 23:24:25,467 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 00:18:22,843 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-04-05 00:18:23,385 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-04-05 00:18:23,386 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VM OVFs in Data Center Lok >2014-04-05 00:18:23,386 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated VM OVFs in Data Center Lok >2014-04-05 00:18:23,386 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update template OVFs in Data Center Lok >2014-04-05 00:18:23,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated templates OVFs in Data Center Lok >2014-04-05 00:18:23,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 00:18:23,388 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 00:24:25,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 70364a63 >2014-04-05 00:24:25,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7c4de322 >2014-04-05 00:24:25,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c4de322 >2014-04-05 00:24:25,838 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 70364a63 >2014-04-05 00:24:25,842 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 01:18:22,845 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-04-05 01:18:23,388 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-04-05 01:18:23,389 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VM OVFs in Data Center Lok >2014-04-05 01:18:23,390 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated VM OVFs in Data Center Lok >2014-04-05 01:18:23,390 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update template OVFs in Data Center Lok >2014-04-05 01:18:23,391 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated templates OVFs in Data Center Lok >2014-04-05 01:18:23,391 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 01:18:23,391 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 01:24:25,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 51fcee63 >2014-04-05 01:24:25,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3486cfa2 >2014-04-05 01:24:26,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3486cfa2 >2014-04-05 01:24:26,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 51fcee63 >2014-04-05 01:24:26,084 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 02:18:22,846 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-04-05 02:18:23,392 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-04-05 02:18:23,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VM OVFs in Data Center Lok >2014-04-05 02:18:23,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated VM OVFs in Data Center Lok >2014-04-05 02:18:23,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update template OVFs in Data Center Lok >2014-04-05 02:18:23,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated templates OVFs in Data Center Lok >2014-04-05 02:18:23,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 02:18:23,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 02:24:26,088 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 71c00956 >2014-04-05 02:24:26,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 76a52771 >2014-04-05 02:24:26,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76a52771 >2014-04-05 02:24:26,336 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 71c00956 >2014-04-05 02:24:26,341 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 03:18:22,847 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-04-05 03:18:23,395 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-04-05 03:18:23,395 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VM OVFs in Data Center Lok >2014-04-05 03:18:23,396 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated VM OVFs in Data Center Lok >2014-04-05 03:18:23,396 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update template OVFs in Data Center Lok >2014-04-05 03:18:23,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated templates OVFs in Data Center Lok >2014-04-05 03:18:23,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 03:18:23,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 03:24:26,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2af6f4b5 >2014-04-05 03:24:26,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 318329d >2014-04-05 03:24:26,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 318329d >2014-04-05 03:24:26,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2af6f4b5 >2014-04-05 03:24:26,844 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-100) Start deleteAgedOutAuditLogs >2014-04-05 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-100) Finished deleteAgedOutAuditLogs >2014-04-05 04:18:22,848 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) Start refreshing all users data >2014-04-05 04:18:23,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-04-05 04:18:23,399 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VM OVFs in Data Center Lok >2014-04-05 04:18:23,400 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated VM OVFs in Data Center Lok >2014-04-05 04:18:23,400 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update template OVFs in Data Center Lok >2014-04-05 04:18:23,401 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated templates OVFs in Data Center Lok >2014-04-05 04:18:23,401 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 04:18:23,401 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 04:24:26,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 34ec6ea7 >2014-04-05 04:24:26,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: e54b116 >2014-04-05 04:24:27,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: e54b116 >2014-04-05 04:24:27,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 34ec6ea7 >2014-04-05 04:24:27,087 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 05:18:22,850 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-05 05:18:23,401 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-05 05:18:23,402 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-04-05 05:18:23,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-04-05 05:18:23,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-04-05 05:18:23,404 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-04-05 05:18:23,404 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 05:18:23,404 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 05:24:27,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4176dcde >2014-04-05 05:24:27,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 629e9465 >2014-04-05 05:24:27,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 629e9465 >2014-04-05 05:24:27,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4176dcde >2014-04-05 05:24:27,329 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 06:18:22,852 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-04-05 06:18:23,404 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-04-05 06:18:23,405 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VM OVFs in Data Center Lok >2014-04-05 06:18:23,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully updated VM OVFs in Data Center Lok >2014-04-05 06:18:23,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update template OVFs in Data Center Lok >2014-04-05 06:18:23,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully updated templates OVFs in Data Center Lok >2014-04-05 06:18:23,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 06:18:23,407 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 06:24:27,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4bbfe01 >2014-04-05 06:24:27,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5220f614 >2014-04-05 06:24:27,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5220f614 >2014-04-05 06:24:27,589 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4bbfe01 >2014-04-05 06:24:27,594 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 07:18:22,854 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-04-05 07:18:23,407 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-04-05 07:18:23,407 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VM OVFs in Data Center Lok >2014-04-05 07:18:23,408 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated VM OVFs in Data Center Lok >2014-04-05 07:18:23,408 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update template OVFs in Data Center Lok >2014-04-05 07:18:23,408 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated templates OVFs in Data Center Lok >2014-04-05 07:18:23,409 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 07:18:23,409 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 07:24:27,597 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5abac8e4 >2014-04-05 07:24:27,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1cdb1f45 >2014-04-05 07:24:27,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1cdb1f45 >2014-04-05 07:24:27,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5abac8e4 >2014-04-05 07:24:27,855 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 08:18:22,855 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-04-05 08:18:23,409 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-04-05 08:18:23,410 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-04-05 08:18:23,410 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-04-05 08:18:23,410 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-04-05 08:18:23,411 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-04-05 08:18:23,411 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 08:18:23,412 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 08:24:27,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1577f58a >2014-04-05 08:24:27,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 76994208 >2014-04-05 08:24:28,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76994208 >2014-04-05 08:24:28,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1577f58a >2014-04-05 08:24:28,102 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 09:18:22,857 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-04-05 09:18:23,412 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-04-05 09:18:23,412 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-04-05 09:18:23,413 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-04-05 09:18:23,413 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-04-05 09:18:23,414 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-04-05 09:18:23,414 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 09:18:23,414 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 09:24:28,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7835dff7 >2014-04-05 09:24:28,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 19030ec9 >2014-04-05 09:24:28,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 19030ec9 >2014-04-05 09:24:28,371 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7835dff7 >2014-04-05 09:24:28,387 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 10:18:22,858 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-04-05 10:18:23,414 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-04-05 10:18:23,415 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VM OVFs in Data Center Lok >2014-04-05 10:18:23,416 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Successfully updated VM OVFs in Data Center Lok >2014-04-05 10:18:23,416 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update template OVFs in Data Center Lok >2014-04-05 10:18:23,417 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Successfully updated templates OVFs in Data Center Lok >2014-04-05 10:18:23,417 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 10:18:23,417 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 10:24:28,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2daff3b2 >2014-04-05 10:24:28,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7aa8428c >2014-04-05 10:24:28,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7aa8428c >2014-04-05 10:24:28,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2daff3b2 >2014-04-05 10:24:28,627 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 11:18:22,860 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-04-05 11:18:23,417 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-04-05 11:18:23,418 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VM OVFs in Data Center Lok >2014-04-05 11:18:23,419 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated VM OVFs in Data Center Lok >2014-04-05 11:18:23,419 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update template OVFs in Data Center Lok >2014-04-05 11:18:23,420 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated templates OVFs in Data Center Lok >2014-04-05 11:18:23,420 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 11:18:23,421 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 11:24:28,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 66c5e31a >2014-04-05 11:24:28,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 49c0f4ac >2014-04-05 11:24:28,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 49c0f4ac >2014-04-05 11:24:28,869 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 66c5e31a >2014-04-05 11:24:28,873 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 12:18:22,863 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-04-05 12:18:23,421 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-04-05 12:18:23,422 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VM OVFs in Data Center Lok >2014-04-05 12:18:23,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated VM OVFs in Data Center Lok >2014-04-05 12:18:23,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update template OVFs in Data Center Lok >2014-04-05 12:18:23,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated templates OVFs in Data Center Lok >2014-04-05 12:18:23,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 12:18:23,424 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 12:24:28,877 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7fdfa994 >2014-04-05 12:24:29,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 433a1165 >2014-04-05 12:24:29,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 433a1165 >2014-04-05 12:24:29,478 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7fdfa994 >2014-04-05 12:24:29,482 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 13:18:22,865 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-04-05 13:18:23,424 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-04-05 13:18:23,425 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VM OVFs in Data Center Lok >2014-04-05 13:18:23,426 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated VM OVFs in Data Center Lok >2014-04-05 13:18:23,426 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update template OVFs in Data Center Lok >2014-04-05 13:18:23,427 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated templates OVFs in Data Center Lok >2014-04-05 13:18:23,427 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 13:18:23,429 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 13:24:29,484 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 23d811f4 >2014-04-05 13:24:29,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3e82bc8e >2014-04-05 13:24:29,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3e82bc8e >2014-04-05 13:24:29,721 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23d811f4 >2014-04-05 13:24:29,726 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 14:18:22,866 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-04-05 14:18:23,429 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-04-05 14:18:23,430 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VM OVFs in Data Center Lok >2014-04-05 14:18:23,431 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated VM OVFs in Data Center Lok >2014-04-05 14:18:23,431 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update template OVFs in Data Center Lok >2014-04-05 14:18:23,431 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated templates OVFs in Data Center Lok >2014-04-05 14:18:23,431 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 14:18:23,432 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 14:24:29,730 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 49526ba4 >2014-04-05 14:24:29,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 42190197 >2014-04-05 14:24:29,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 42190197 >2014-04-05 14:24:29,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 49526ba4 >2014-04-05 14:24:30,000 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 15:18:22,868 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-50) Start refreshing all users data >2014-04-05 15:18:23,432 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-04-05 15:18:23,433 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VM OVFs in Data Center Lok >2014-04-05 15:18:23,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated VM OVFs in Data Center Lok >2014-04-05 15:18:23,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update template OVFs in Data Center Lok >2014-04-05 15:18:23,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated templates OVFs in Data Center Lok >2014-04-05 15:18:23,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 15:18:23,435 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 15:24:30,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 76314478 >2014-04-05 15:24:30,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1c865ee7 >2014-04-05 15:24:30,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1c865ee7 >2014-04-05 15:24:30,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76314478 >2014-04-05 15:24:30,259 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 16:18:22,870 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-5) Start refreshing all users data >2014-04-05 16:18:23,435 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-04-05 16:18:23,436 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-04-05 16:18:23,437 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-04-05 16:18:23,437 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-04-05 16:18:23,437 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-04-05 16:18:23,437 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 16:18:23,438 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 16:24:30,262 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d2b71de >2014-04-05 16:24:30,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: f40b4c8 >2014-04-05 16:24:30,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: f40b4c8 >2014-04-05 16:24:30,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d2b71de >2014-04-05 16:24:30,710 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 17:18:22,872 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-04-05 17:18:23,438 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-04-05 17:18:23,439 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VM OVFs in Data Center Lok >2014-04-05 17:18:23,439 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated VM OVFs in Data Center Lok >2014-04-05 17:18:23,439 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update template OVFs in Data Center Lok >2014-04-05 17:18:23,440 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated templates OVFs in Data Center Lok >2014-04-05 17:18:23,440 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 17:18:23,440 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 17:24:30,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 32f420df >2014-04-05 17:24:30,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 399a2a9b >2014-04-05 17:24:30,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 399a2a9b >2014-04-05 17:24:30,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 32f420df >2014-04-05 17:24:30,970 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 18:18:22,875 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-04-05 18:18:23,440 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-05 18:18:23,441 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-05 18:18:23,441 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-05 18:18:23,442 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-05 18:18:23,442 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-05 18:18:23,442 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 18:18:23,443 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 18:24:30,974 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 29f59e1a >2014-04-05 18:24:30,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3c5ed27b >2014-04-05 18:24:31,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c5ed27b >2014-04-05 18:24:31,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 29f59e1a >2014-04-05 18:24:31,230 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 19:18:22,876 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-04-05 19:18:23,443 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-04-05 19:18:23,444 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VM OVFs in Data Center Lok >2014-04-05 19:18:23,445 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated VM OVFs in Data Center Lok >2014-04-05 19:18:23,445 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update template OVFs in Data Center Lok >2014-04-05 19:18:23,445 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated templates OVFs in Data Center Lok >2014-04-05 19:18:23,445 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 19:18:23,446 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 19:24:31,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 528924e4 >2014-04-05 19:24:31,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 29d51f11 >2014-04-05 19:24:31,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 29d51f11 >2014-04-05 19:24:31,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 528924e4 >2014-04-05 19:24:31,491 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 20:18:22,877 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-04-05 20:18:23,447 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-04-05 20:18:23,448 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VM OVFs in Data Center Lok >2014-04-05 20:18:23,448 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated VM OVFs in Data Center Lok >2014-04-05 20:18:23,448 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update template OVFs in Data Center Lok >2014-04-05 20:18:23,449 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated templates OVFs in Data Center Lok >2014-04-05 20:18:23,449 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 20:18:23,450 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 20:24:31,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 713ff1c0 >2014-04-05 20:24:31,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1d18ae56 >2014-04-05 20:24:31,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d18ae56 >2014-04-05 20:24:31,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 713ff1c0 >2014-04-05 20:24:31,757 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 21:18:22,878 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-04-05 21:18:23,450 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-04-05 21:18:23,451 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VM OVFs in Data Center Lok >2014-04-05 21:18:23,452 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated VM OVFs in Data Center Lok >2014-04-05 21:18:23,452 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update template OVFs in Data Center Lok >2014-04-05 21:18:23,453 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated templates OVFs in Data Center Lok >2014-04-05 21:18:23,453 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 21:18:23,453 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 21:24:31,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 64751505 >2014-04-05 21:24:31,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 42b72ec9 >2014-04-05 21:24:31,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 42b72ec9 >2014-04-05 21:24:31,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 64751505 >2014-04-05 21:24:32,000 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 22:18:22,881 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) Start refreshing all users data >2014-04-05 22:18:23,454 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-05 22:18:23,455 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-05 22:18:23,456 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-05 22:18:23,456 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-05 22:18:23,461 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-05 22:18:23,461 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 22:18:23,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 22:24:32,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2d8680e0 >2014-04-05 22:24:32,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 59770979 >2014-04-05 22:24:32,239 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 59770979 >2014-04-05 22:24:32,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d8680e0 >2014-04-05 22:24:32,244 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-05 23:18:22,882 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-04-05 23:18:23,463 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-04-05 23:18:23,464 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VM OVFs in Data Center Lok >2014-04-05 23:18:23,464 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated VM OVFs in Data Center Lok >2014-04-05 23:18:23,464 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update template OVFs in Data Center Lok >2014-04-05 23:18:23,465 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated templates OVFs in Data Center Lok >2014-04-05 23:18:23,465 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-05 23:18:23,465 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-05 23:24:32,248 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 13594477 >2014-04-05 23:24:32,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 499dea8d >2014-04-05 23:24:32,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 499dea8d >2014-04-05 23:24:32,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 13594477 >2014-04-05 23:24:32,480 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 00:18:22,881 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-04-06 00:18:23,466 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-04-06 00:18:23,467 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VM OVFs in Data Center Lok >2014-04-06 00:18:23,468 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated VM OVFs in Data Center Lok >2014-04-06 00:18:23,469 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update template OVFs in Data Center Lok >2014-04-06 00:18:23,469 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated templates OVFs in Data Center Lok >2014-04-06 00:18:23,469 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 00:18:23,470 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 00:24:32,486 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 79dcfda1 >2014-04-06 00:24:32,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1b8f9388 >2014-04-06 00:24:32,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b8f9388 >2014-04-06 00:24:32,722 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 79dcfda1 >2014-04-06 00:24:32,727 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 01:18:22,882 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-04-06 01:18:23,471 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-06 01:18:23,472 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-04-06 01:18:23,473 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-04-06 01:18:23,473 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-04-06 01:18:23,474 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-04-06 01:18:23,474 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 01:18:23,474 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 01:24:32,731 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 55e0c7f6 >2014-04-06 01:24:32,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 18681386 >2014-04-06 01:24:32,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 18681386 >2014-04-06 01:24:32,964 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 55e0c7f6 >2014-04-06 01:24:32,969 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 02:18:22,884 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) Start refreshing all users data >2014-04-06 02:18:23,475 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-04-06 02:18:23,476 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Lok >2014-04-06 02:18:23,476 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Lok >2014-04-06 02:18:23,476 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Lok >2014-04-06 02:18:23,477 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center Lok >2014-04-06 02:18:23,477 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 02:18:23,477 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 02:24:32,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4f0b56d7 >2014-04-06 02:24:32,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 717265cb >2014-04-06 02:24:33,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 717265cb >2014-04-06 02:24:33,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f0b56d7 >2014-04-06 02:24:33,211 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 03:18:22,885 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-04-06 03:18:23,477 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-04-06 03:18:23,478 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-04-06 03:18:23,479 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-04-06 03:18:23,479 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-04-06 03:18:23,480 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-04-06 03:18:23,480 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 03:18:23,480 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 03:24:33,214 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2b94fcf8 >2014-04-06 03:24:33,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 426c1422 >2014-04-06 03:24:33,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 426c1422 >2014-04-06 03:24:33,587 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2b94fcf8 >2014-04-06 03:24:33,591 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-32) Start deleteAgedOutAuditLogs >2014-04-06 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-32) Finished deleteAgedOutAuditLogs >2014-04-06 04:18:22,887 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-04-06 04:18:23,481 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-04-06 04:18:23,482 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VM OVFs in Data Center Lok >2014-04-06 04:18:23,483 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated VM OVFs in Data Center Lok >2014-04-06 04:18:23,483 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update template OVFs in Data Center Lok >2014-04-06 04:18:23,484 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated templates OVFs in Data Center Lok >2014-04-06 04:18:23,484 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 04:18:23,485 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 04:24:33,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6df2f1fa >2014-04-06 04:24:33,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 68ec8c7a >2014-04-06 04:24:38,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 68ec8c7a >2014-04-06 04:24:38,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6df2f1fa >2014-04-06 04:24:38,962 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 05:18:22,888 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-04-06 05:18:23,485 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-04-06 05:18:23,486 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VM OVFs in Data Center Lok >2014-04-06 05:18:23,487 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated VM OVFs in Data Center Lok >2014-04-06 05:18:23,487 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update template OVFs in Data Center Lok >2014-04-06 05:18:23,487 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated templates OVFs in Data Center Lok >2014-04-06 05:18:23,487 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 05:18:23,488 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 05:24:38,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6bdce6b3 >2014-04-06 05:24:46,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 444e5fb6 >2014-04-06 05:24:51,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 444e5fb6 >2014-04-06 05:24:51,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6bdce6b3 >2014-04-06 05:24:51,405 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 06:18:22,889 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-42) Start refreshing all users data >2014-04-06 06:18:23,488 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-04-06 06:18:23,489 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VM OVFs in Data Center Lok >2014-04-06 06:18:23,490 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated VM OVFs in Data Center Lok >2014-04-06 06:18:23,490 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update template OVFs in Data Center Lok >2014-04-06 06:18:23,491 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated templates OVFs in Data Center Lok >2014-04-06 06:18:23,491 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 06:18:23,491 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 06:24:51,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 348dcfa >2014-04-06 06:24:51,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 24a3a2c5 >2014-04-06 06:24:56,770 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 24a3a2c5 >2014-04-06 06:24:56,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 348dcfa >2014-04-06 06:24:56,777 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 07:18:22,891 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-04-06 07:18:23,492 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-06 07:18:23,493 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-06 07:18:23,493 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-06 07:18:23,494 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-06 07:18:23,494 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-06 07:18:23,494 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 07:18:23,495 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 07:24:56,781 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5c71ad21 >2014-04-06 07:24:58,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7bba65cf >2014-04-06 07:25:04,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7bba65cf >2014-04-06 07:25:04,231 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5c71ad21 >2014-04-06 07:25:04,235 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 08:18:22,892 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-04-06 08:18:23,496 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-04-06 08:18:23,497 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VM OVFs in Data Center Lok >2014-04-06 08:18:23,497 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated VM OVFs in Data Center Lok >2014-04-06 08:18:23,497 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update template OVFs in Data Center Lok >2014-04-06 08:18:23,498 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated templates OVFs in Data Center Lok >2014-04-06 08:18:23,498 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 08:18:23,499 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 08:25:04,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5a3f94b9 >2014-04-06 08:25:12,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 904c397 >2014-04-06 08:25:17,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 904c397 >2014-04-06 08:25:17,924 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a3f94b9 >2014-04-06 08:25:17,929 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 09:18:22,893 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-04-06 09:18:23,499 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-04-06 09:18:23,500 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VM OVFs in Data Center Lok >2014-04-06 09:18:23,501 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated VM OVFs in Data Center Lok >2014-04-06 09:18:23,501 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update template OVFs in Data Center Lok >2014-04-06 09:18:23,501 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated templates OVFs in Data Center Lok >2014-04-06 09:18:23,501 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 09:18:23,502 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 09:25:17,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4ee6046d >2014-04-06 09:25:22,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 415f7e6a >2014-04-06 09:25:27,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 415f7e6a >2014-04-06 09:25:27,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4ee6046d >2014-04-06 09:25:27,750 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 10:18:22,894 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-04-06 10:18:23,502 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-04-06 10:18:23,503 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VM OVFs in Data Center Lok >2014-04-06 10:18:23,504 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated VM OVFs in Data Center Lok >2014-04-06 10:18:23,504 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update template OVFs in Data Center Lok >2014-04-06 10:18:23,504 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated templates OVFs in Data Center Lok >2014-04-06 10:18:23,504 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 10:18:23,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 10:25:27,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 650c71bd >2014-04-06 10:25:27,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 13ff631f >2014-04-06 10:25:33,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 13ff631f >2014-04-06 10:25:33,119 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 650c71bd >2014-04-06 10:25:33,126 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 11:18:22,897 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-63) Start refreshing all users data >2014-04-06 11:18:23,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-04-06 11:18:23,506 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VM OVFs in Data Center Lok >2014-04-06 11:18:23,507 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully updated VM OVFs in Data Center Lok >2014-04-06 11:18:23,507 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update template OVFs in Data Center Lok >2014-04-06 11:18:23,507 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully updated templates OVFs in Data Center Lok >2014-04-06 11:18:23,507 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 11:18:23,508 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 11:25:33,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5e04cb36 >2014-04-06 11:25:34,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 752b5193 >2014-04-06 11:25:39,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 752b5193 >2014-04-06 11:25:39,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e04cb36 >2014-04-06 11:25:39,604 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 12:18:22,899 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-04-06 12:18:23,509 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-04-06 12:18:23,509 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VM OVFs in Data Center Lok >2014-04-06 12:18:23,510 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated VM OVFs in Data Center Lok >2014-04-06 12:18:23,510 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update template OVFs in Data Center Lok >2014-04-06 12:18:23,510 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated templates OVFs in Data Center Lok >2014-04-06 12:18:23,510 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 12:18:23,511 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 12:25:39,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7a98045d >2014-04-06 12:25:50,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 757717c >2014-04-06 12:25:55,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 757717c >2014-04-06 12:25:55,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7a98045d >2014-04-06 12:25:55,705 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 13:18:22,900 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-04-06 13:18:23,512 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-06 13:18:23,513 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-06 13:18:23,513 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-06 13:18:23,513 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-06 13:18:23,514 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-06 13:18:23,514 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 13:18:23,514 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 13:25:55,708 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 588c3db3 >2014-04-06 13:25:55,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6bb517cd >2014-04-06 13:25:56,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6bb517cd >2014-04-06 13:25:56,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 588c3db3 >2014-04-06 13:25:56,050 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 14:18:22,901 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-04-06 14:18:23,515 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-04-06 14:18:23,516 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VM OVFs in Data Center Lok >2014-04-06 14:18:23,516 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated VM OVFs in Data Center Lok >2014-04-06 14:18:23,516 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update template OVFs in Data Center Lok >2014-04-06 14:18:23,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated templates OVFs in Data Center Lok >2014-04-06 14:18:23,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 14:18:23,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 14:25:56,053 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [6523099e] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7285da6f >2014-04-06 14:25:56,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [6523099e] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 31c7e826 >2014-04-06 14:25:56,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [6523099e] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 31c7e826 >2014-04-06 14:25:56,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [6523099e] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7285da6f >2014-04-06 14:25:56,289 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) [6523099e] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 15:18:22,902 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-04-06 15:18:23,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-06 15:18:23,518 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-06 15:18:23,519 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-06 15:18:23,519 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-06 15:18:23,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-06 15:18:23,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 15:18:23,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 15:25:56,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 567c0427 >2014-04-06 15:25:56,293 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 67f4c8cc >2014-04-06 15:25:56,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 67f4c8cc >2014-04-06 15:25:56,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 567c0427 >2014-04-06 15:25:56,530 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 16:18:22,903 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-04-06 16:18:23,521 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-04-06 16:18:23,522 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VM OVFs in Data Center Lok >2014-04-06 16:18:23,523 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully updated VM OVFs in Data Center Lok >2014-04-06 16:18:23,523 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update template OVFs in Data Center Lok >2014-04-06 16:18:23,524 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully updated templates OVFs in Data Center Lok >2014-04-06 16:18:23,524 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 16:18:23,525 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 16:25:56,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4353fcfb >2014-04-06 16:25:56,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3f05e545 >2014-04-06 16:25:56,765 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3f05e545 >2014-04-06 16:25:56,765 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4353fcfb >2014-04-06 16:25:56,773 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 17:18:22,905 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-04-06 17:18:23,525 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-04-06 17:18:23,526 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VM OVFs in Data Center Lok >2014-04-06 17:18:23,527 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated VM OVFs in Data Center Lok >2014-04-06 17:18:23,527 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update template OVFs in Data Center Lok >2014-04-06 17:18:23,527 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated templates OVFs in Data Center Lok >2014-04-06 17:18:23,527 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 17:18:23,528 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 17:25:56,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3dbe1148 >2014-04-06 17:25:56,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 960ada3 >2014-04-06 17:25:57,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 960ada3 >2014-04-06 17:25:57,009 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3dbe1148 >2014-04-06 17:25:57,015 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 18:18:22,905 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-18) Start refreshing all users data >2014-04-06 18:18:23,528 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-04-06 18:18:23,529 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VM OVFs in Data Center Lok >2014-04-06 18:18:23,530 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated VM OVFs in Data Center Lok >2014-04-06 18:18:23,530 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update template OVFs in Data Center Lok >2014-04-06 18:18:23,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated templates OVFs in Data Center Lok >2014-04-06 18:18:23,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 18:18:23,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 18:25:57,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 58a175af >2014-04-06 18:25:57,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5035619e >2014-04-06 18:25:57,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5035619e >2014-04-06 18:25:57,253 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 58a175af >2014-04-06 18:25:57,257 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 19:18:22,908 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-04-06 19:18:23,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-04-06 19:18:23,532 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VM OVFs in Data Center Lok >2014-04-06 19:18:23,533 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated VM OVFs in Data Center Lok >2014-04-06 19:18:23,533 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update template OVFs in Data Center Lok >2014-04-06 19:18:23,534 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated templates OVFs in Data Center Lok >2014-04-06 19:18:23,534 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 19:18:23,534 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 19:25:57,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 40f9c659 >2014-04-06 19:25:57,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4e4d1e62 >2014-04-06 19:25:57,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4e4d1e62 >2014-04-06 19:25:57,493 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 40f9c659 >2014-04-06 19:25:57,498 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 20:18:22,910 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-04-06 20:18:23,534 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-04-06 20:18:23,536 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VM OVFs in Data Center Lok >2014-04-06 20:18:23,537 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated VM OVFs in Data Center Lok >2014-04-06 20:18:23,537 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update template OVFs in Data Center Lok >2014-04-06 20:18:23,537 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated templates OVFs in Data Center Lok >2014-04-06 20:18:23,537 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 20:18:23,538 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 20:25:57,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4a9257e2 >2014-04-06 20:25:57,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4dc7dcad >2014-04-06 20:25:57,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4dc7dcad >2014-04-06 20:25:57,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4a9257e2 >2014-04-06 20:25:57,744 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 21:18:22,912 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-04-06 21:18:23,538 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-04-06 21:18:23,539 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VM OVFs in Data Center Lok >2014-04-06 21:18:23,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated VM OVFs in Data Center Lok >2014-04-06 21:18:23,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update template OVFs in Data Center Lok >2014-04-06 21:18:23,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated templates OVFs in Data Center Lok >2014-04-06 21:18:23,541 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 21:18:23,541 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 21:25:57,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 399eae81 >2014-04-06 21:25:57,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 60d2b4b4 >2014-04-06 21:25:58,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 60d2b4b4 >2014-04-06 21:25:58,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 399eae81 >2014-04-06 21:25:58,172 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 22:18:22,914 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-86) Start refreshing all users data >2014-04-06 22:18:23,541 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-06 22:18:23,542 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-06 22:18:23,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-06 22:18:23,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-06 22:18:23,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-06 22:18:23,544 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 22:18:23,544 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 22:25:58,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 25403640 >2014-04-06 22:25:58,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1aa9b6db >2014-04-06 22:25:58,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1aa9b6db >2014-04-06 22:25:58,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 25403640 >2014-04-06 22:25:58,423 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-06 23:18:22,916 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-06 23:18:23,544 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-06 23:18:23,545 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-06 23:18:23,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-06 23:18:23,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-06 23:18:23,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-06 23:18:23,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-06 23:18:23,547 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-06 23:25:58,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 53e0f17f >2014-04-06 23:25:58,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 550ef3de >2014-04-06 23:25:58,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 550ef3de >2014-04-06 23:25:58,716 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 53e0f17f >2014-04-06 23:25:58,720 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 00:18:22,917 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-04-07 00:18:23,547 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-07 00:18:23,548 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-07 00:18:23,550 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-07 00:18:23,550 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-07 00:18:23,551 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-07 00:18:23,551 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 00:18:23,552 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 00:25:58,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4660c676 >2014-04-07 00:25:58,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7b859112 >2014-04-07 00:25:58,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7b859112 >2014-04-07 00:25:58,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4660c676 >2014-04-07 00:25:58,988 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 01:18:22,919 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) Start refreshing all users data >2014-04-07 01:18:23,552 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-04-07 01:18:23,553 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VM OVFs in Data Center Lok >2014-04-07 01:18:23,554 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated VM OVFs in Data Center Lok >2014-04-07 01:18:23,554 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update template OVFs in Data Center Lok >2014-04-07 01:18:23,555 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated templates OVFs in Data Center Lok >2014-04-07 01:18:23,555 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 01:18:23,555 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 01:25:58,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 8d45a78 >2014-04-07 01:25:58,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2d7f80cf >2014-04-07 01:25:59,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d7f80cf >2014-04-07 01:25:59,257 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 8d45a78 >2014-04-07 01:25:59,262 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 02:18:22,921 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-04-07 02:18:23,556 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-04-07 02:18:23,557 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VM OVFs in Data Center Lok >2014-04-07 02:18:23,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated VM OVFs in Data Center Lok >2014-04-07 02:18:23,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update template OVFs in Data Center Lok >2014-04-07 02:18:23,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated templates OVFs in Data Center Lok >2014-04-07 02:18:23,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 02:18:23,559 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 02:25:59,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1e4ee382 >2014-04-07 02:25:59,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4d3a7b59 >2014-04-07 02:25:59,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4d3a7b59 >2014-04-07 02:25:59,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1e4ee382 >2014-04-07 02:25:59,688 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 09:20:49,999 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-04-07 09:20:50,000 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Loaded file "/etc/ovirt-engine/engine.conf". >2014-04-07 09:20:50,000 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-04-07 09:20:50,001 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-04-07 09:20:50,001 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-04-07 09:20:50,001 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-04-07 09:20:50,001 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-04-07 09:20:50,001 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-04-07 09:20:50,002 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-04-07 09:20:50,020 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_AJP_PORT" is "8702". >2014-04-07 09:20:50,020 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_APPS" is "engine.ear redhat-support-plugin-rhev.ear". >2014-04-07 09:20:50,020 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-04-07 09:20:50,020 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-04-07 09:20:50,020 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-04-07 09:20:50,020 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-04-07 09:20:50,020 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-04-07 09:20:50,020 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_HOST" is "localhost". >2014-04-07 09:20:50,020 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_PORT" is "5432". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_SECURED" is "False". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_SSL" is "false". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DB_USER" is "engine". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_FQDN" is "localhost.localdomain". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_GROUP" is "ovirt". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-04-07 09:20:50,021 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-04-07 09:20:50,022 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-04-07 09:20:50,022 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-04-07 09:20:50,022 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-04-07 09:20:50,022 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_HTTP_PORT" is "None". >2014-04-07 09:20:50,031 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-04-07 09:20:50,031 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-04-07 09:20:50,031 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-04-07 09:20:50,031 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-04-07 09:20:50,031 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PERM_MAX" is "256m". >2014-04-07 09:20:50,031 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PERM_MIN" is "256m". >2014-04-07 09:20:50,031 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-04-07 09:20:50,031 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-04-07 09:20:50,031 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-04-07 09:20:50,031 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_STOP_TIME" is "10". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_USER" is "ovirt". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-04-07 09:20:50,032 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-04-07 09:20:50,033 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-04-07 09:20:50,033 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 36) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-04-07 09:20:51,056 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 41) Start initializing Backend >2014-04-07 09:20:51,507 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ServerService Thread Pool -- 41) Failed to decrypt value for property LocalAdminPassword will be used encrypted value: javax.crypto.BadPaddingException: Data must start with zero > at sun.security.rsa.RSAPadding.unpadV15(RSAPadding.java:329) [rt.jar:1.7.0_51] > at sun.security.rsa.RSAPadding.unpad(RSAPadding.java:272) [rt.jar:1.7.0_51] > at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:356) [sunjce_provider.jar:1.7.0_51] > at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:382) [sunjce_provider.jar:1.7.0_51] > at javax.crypto.Cipher.doFinal(Cipher.java:1922) [jce.jar:1.7.0_51] > at org.ovirt.engine.core.utils.crypt.EngineEncryptionUtils.decrypt(EngineEncryptionUtils.java:169) [utils.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.getValue(DBConfigUtils.java:107) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.updateOption(DBConfigUtils.java:275) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.refreshVdcOptionCache(DBConfigUtils.java:35) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.<init>(DBConfigUtils.java:166) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.<init>(DBConfigUtils.java:152) [dal.jar:] > at org.ovirt.engine.core.bll.Backend.Initialize(Backend.java:183) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.create(Backend.java:130) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:130) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:125) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:140) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > at org.jboss.threads.JBossThread.run(JBossThread.java:122) > >2014-04-07 09:20:51,655 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 41) Could not find enum value for option: WebAdminUpperRightButtonURL >2014-04-07 09:20:51,750 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 41) Running ovirt-engine 3.3.1-2.BZ1070265.el6ev >2014-04-07 09:20:51,755 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 41) Start initializing dictionaries >2014-04-07 09:20:51,764 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 41) Finished initializing dictionaries >2014-04-07 09:20:51,764 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 41) Start initializing AuditLogCleanupManager >2014-04-07 09:20:51,765 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 41) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-04-07 09:20:51,809 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 41) Finished initializing AuditLogCleanupManager >2014-04-07 09:20:51,819 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 41) Start initializing TagsDirector >2014-04-07 09:20:51,842 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 41) Tag root added to tree >2014-04-07 09:20:51,872 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 41) Finished initializing TagsDirector >2014-04-07 09:20:51,876 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 41) Start initializing IsoDomainListSyncronizer >2014-04-07 09:20:51,891 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 41) Finished initializing IsoDomainListSyncronizer >2014-04-07 09:20:51,906 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 41) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-04-07 09:20:52,225 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 41) Completed initializing handlers >2014-04-07 09:20:52,226 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 41) Start initializing ErrorTranslatorImpl >2014-04-07 09:20:52,252 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 41) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-04-07 09:20:52,253 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 41) Finished initializing ErrorTranslatorImpl >2014-04-07 09:20:52,253 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 41) Start initializing ErrorTranslatorImpl >2014-04-07 09:20:52,254 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 41) Finished initializing ErrorTranslatorImpl >2014-04-07 09:20:52,255 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 41) Mark incomplete jobs as UNKNOWN >2014-04-07 09:20:52,303 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 41) Start initializing JobRepositoryCleanupManager >2014-04-07 09:20:52,304 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 41) Finished initializing JobRepositoryCleanupManager >2014-04-07 09:20:52,306 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 41) Start initializing AutoRecoveryManager >2014-04-07 09:20:52,310 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 41) Finished initializing AutoRecoveryManager >2014-04-07 09:20:52,312 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 41) Start initializing ExecutionMessageDirector >2014-04-07 09:20:52,321 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 41) Finished initializing ExecutionMessageDirector >2014-04-07 09:20:52,357 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 38) Start initializing UsersDomainsCacheManagerService >2014-04-07 09:20:52,380 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (ServerService Thread Pool -- 38) Start initializing DbUserCacheManager >2014-04-07 09:20:52,388 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (ServerService Thread Pool -- 38) Finished initializing DbUserCacheManager >2014-04-07 09:20:52,388 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 38) Finished initializing UsersDomainsCacheManagerService >2014-04-07 09:20:52,403 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ServerService Thread Pool -- 39) Initialization of AsyncTaskManager completed successfully. >2014-04-07 09:20:52,410 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 39) Start initializing ResourceManager >2014-04-07 09:20:52,434 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-04-07 09:20:52,664 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 39) Entered VdsManager constructor >2014-04-07 09:20:52,715 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 39) Initialize vdsBroker (10.34.27.209,54,321) >2014-04-07 09:20:52,980 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 39) VDS 53fd133a-9dc6-11e2-abd9-001a4a223991 was added to the Resource Manager >2014-04-07 09:20:53,012 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 39) Finished initializing ResourceManager >2014-04-07 09:20:53,019 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 39) Initialization of OvfDataUpdater completed successfully. >2014-04-07 09:20:53,025 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 39) Start scheduling to enable vds load balancer >2014-04-07 09:20:53,027 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 39) Finished scheduling to enable vds load balancer >2014-04-07 09:20:53,045 INFO [org.ovirt.engine.core.bll.network.MacPoolManager] (pool-4-thread-1) MacPoolManager(2a745455): Start initializing >2014-04-07 09:20:53,072 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 39) Init VM custom properties utilities >2014-04-07 09:20:53,073 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 39) Init device custom properties utilities >2014-04-07 09:20:53,084 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 39) Initializing Scheduling manager >2014-04-07 09:20:53,142 INFO [org.ovirt.engine.core.bll.network.MacPoolManager] (pool-4-thread-1) MacPoolManager(2a745455): Finished initializing >2014-04-07 09:20:53,166 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 39) External scheduler disabled, discovery skipped >2014-04-07 09:20:53,166 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 39) Initialized Scheduling manager >2014-04-07 09:20:53,167 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 39) Initializing DWH Heart Beat >2014-04-07 09:20:57,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1cdc8b13 >2014-04-07 09:20:57,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, GetHardwareInfoVDSCommand, log id: 1cdc8b13 >2014-04-07 09:20:57,883 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: UNASSIGNED not exist in string table >2014-04-07 09:20:57,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-04-07 09:20:57,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-04-07 09:20:57,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-04-07 09:20:57,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-04-07 09:20:57,885 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: DWH_STOPPED not exist in string table >2014-04-07 09:20:57,885 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: DWH_STARTED not exist in string table >2014-04-07 09:20:57,885 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: DWH_ERROR not exist in string table >2014-04-07 09:20:57,917 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_FAILED_REMOVE_VM not have severity. Assumed Normal >2014-04-07 09:20:57,917 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2014-04-07 09:20:57,917 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2014-04-07 09:20:57,917 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_FAILED_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2014-04-07 09:20:57,918 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not have severity. Assumed Normal >2014-04-07 09:20:57,918 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: GLUSTER_VOLUME_OPTION_CHANGED_FROM_CLI not have severity. Assumed Normal >2014-04-07 09:20:57,918 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: GLUSTER_SERVICES_LIST_NOT_FETCHED not have severity. Assumed Normal >2014-04-07 09:20:57,924 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: GLUSTER_VOLUME_BRICK_ADDED not have severity. Assumed Normal >2014-04-07 09:20:57,924 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_EXTEND_DISK_SIZE_UPDATE_VM_FAILURE not have severity. Assumed Normal >2014-04-07 09:20:57,924 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: VM_MIGRATION_START_SYSTEM_INITIATED not have severity. Assumed Normal >2014-04-07 09:20:57,924 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not have severity. Assumed Normal >2014-04-07 09:20:57,924 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_UP not have severity. Assumed Normal >2014-04-07 09:20:57,924 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_DOWN not have severity. Assumed Normal >2014-04-07 09:20:57,925 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_UPDATED_MOM_POLICIES not have severity. Assumed Normal >2014-04-07 09:20:57,925 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_FAILED_TO_UPDATE_MOM_POLICIES not have severity. Assumed Normal >2014-04-07 09:20:58,015 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-07 09:20:58,413 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Initializing Host: 10.34.27.209 >2014-04-07 09:20:58,518 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-3) [3fd33352] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-07 09:21:01,635 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-6) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-04-07 09:21:01,668 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-6) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-04-07 09:21:03,068 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-8) [4209317e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-07 09:21:03,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [4209317e] Correlation ID: 4209317e, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Lok. Setting status to Non Responsive. >2014-04-07 09:21:04,018 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-8) [4209317e] hostFromVds::selectedVds - 10.34.27.209, spmStatus SPM, storage pool Lok >2014-04-07 09:21:04,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-8) [4209317e] Initialize Irs proxy from vds: 10.34.27.209 >2014-04-07 09:21:04,047 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [4209317e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host 10.34.27.209 (Address: 10.34.27.209). >2014-04-07 09:21:04,236 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-10) [4209317e] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2897a0dc >2014-04-07 09:21:04,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-10) [4209317e] -- executeIrsBrokerCommand: Attempting on storage pool 19dcb0b3-4c52-4803-a39e-992b1392da62 >2014-04-07 09:21:04,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (pool-4-thread-10) [4209317e] START, HSMGetAllTasksInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 777ad1b2 >2014-04-07 09:21:04,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (pool-4-thread-10) [4209317e] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 777ad1b2 >2014-04-07 09:21:04,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-10) [4209317e] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 2897a0dc >2014-04-07 09:21:04,652 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-10) [4209317e] Discovered no tasks on Storage Pool Lok >2014-04-07 09:21:09,674 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginAdminUserCommand internal: false. >2014-04-07 09:21:09,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-07 09:21:13,068 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-04-07 09:21:13,091 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-07 09:23:45,325 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2014-04-07 09:23:45,345 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-07 09:23:48,261 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-04-07 09:23:48,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-07 09:23:48,409 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [45871141] Running command: LogoutUserCommand internal: false. >2014-04-07 09:23:48,424 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [45871141] Correlation ID: 45871141, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged out. >2014-04-07 09:24:42,757 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-7) Running command: LoginUserCommand internal: false. >2014-04-07 09:24:42,774 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-07 09:24:42,836 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-7) [3efd1d91] Running command: LogoutUserCommand internal: false. >2014-04-07 09:24:42,855 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) [3efd1d91] Correlation ID: 3efd1d91, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged out. >2014-04-07 09:24:45,413 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-04-07 09:24:45,424 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-07 09:24:45,508 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [50d4e7fb] Running command: LogoutUserCommand internal: false. >2014-04-07 09:24:45,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [50d4e7fb] Correlation ID: 50d4e7fb, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged out. >2014-04-07 09:25:51,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 11583b6b >2014-04-07 09:25:51,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 22b6b257 >2014-04-07 09:25:52,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 22b6b257 >2014-04-07 09:25:52,159 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 11583b6b >2014-04-07 09:25:52,182 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 09:28:29,222 ERROR [org.ovirt.engine.core.utils.servlet.ServletUtils] (ajp-/127.0.0.1:8702-9) Can't read file "/usr/share/doc/rhevm-doc/manual/en-US/html/Administration_Guide/Common_Content/css/lang.css" for request "/docs/en-US/html/Administration_Guide/Common_Content/css/lang.css", will send a 404 error response. >2014-04-07 09:28:52,021 ERROR [org.ovirt.engine.core.utils.servlet.ServletUtils] (ajp-/127.0.0.1:8702-8) Can't read file "/usr/share/doc/rhevm-doc/manual/en-US/html/Administration_Guide/Common_Content/css/lang.css" for request "/docs/en-US/html/Administration_Guide/Common_Content/css/lang.css", will send a 404 error response. >2014-04-07 09:29:58,937 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-04-07 09:29:58,944 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-07 09:29:58,986 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-4) [4a1407dc] Running command: LogoutUserCommand internal: false. >2014-04-07 09:29:58,993 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [4a1407dc] Correlation ID: 4a1407dc, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged out. >2014-04-07 09:30:01,460 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-3) Running command: LoginUserCommand internal: false. >2014-04-07 09:30:01,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-07 09:30:01,518 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-3) [4f16192c] Running command: LogoutUserCommand internal: false. >2014-04-07 09:30:01,530 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [4f16192c] Correlation ID: 4f16192c, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged out. >2014-04-07 09:30:01,710 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-11) Running command: LoginUserCommand internal: false. >2014-04-07 09:30:01,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-07 09:30:01,784 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-11) [2f9113de] Running command: LogoutUserCommand internal: false. >2014-04-07 09:30:01,788 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-11) [2f9113de] Correlation ID: 2f9113de, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged out. >2014-04-07 09:30:44,784 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-5) Running command: LoginUserCommand internal: false. >2014-04-07 09:30:44,797 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-07 09:30:44,841 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-5) [6d456016] Running command: LogoutUserCommand internal: false. >2014-04-07 09:30:44,850 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [6d456016] Correlation ID: 6d456016, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged out. >2014-04-07 10:20:52,461 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-04-07 10:20:53,018 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-04-07 10:20:53,028 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VM OVFs in Data Center Lok >2014-04-07 10:20:53,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully updated VM OVFs in Data Center Lok >2014-04-07 10:20:53,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update template OVFs in Data Center Lok >2014-04-07 10:20:53,046 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully updated templates OVFs in Data Center Lok >2014-04-07 10:20:53,046 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 10:20:53,049 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 10:25:52,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4876d519 >2014-04-07 10:25:52,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 60946936 >2014-04-07 10:25:52,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 60946936 >2014-04-07 10:25:52,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4876d519 >2014-04-07 10:25:52,616 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 11:20:52,462 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-04-07 11:20:53,050 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-04-07 11:20:53,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VM OVFs in Data Center Lok >2014-04-07 11:20:53,053 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated VM OVFs in Data Center Lok >2014-04-07 11:20:53,053 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update template OVFs in Data Center Lok >2014-04-07 11:20:53,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated templates OVFs in Data Center Lok >2014-04-07 11:20:53,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 11:20:53,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 11:25:52,620 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 479a8194 >2014-04-07 11:25:52,621 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 70b14045 >2014-04-07 11:25:53,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 70b14045 >2014-04-07 11:25:53,032 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 479a8194 >2014-04-07 11:25:53,037 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 12:20:52,465 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-04-07 12:20:53,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-04-07 12:20:53,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Lok >2014-04-07 12:20:53,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Lok >2014-04-07 12:20:53,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Lok >2014-04-07 12:20:53,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated templates OVFs in Data Center Lok >2014-04-07 12:20:53,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 12:20:53,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 12:25:53,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7afdb2d6 >2014-04-07 12:25:53,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 25e83eaf >2014-04-07 12:25:53,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 25e83eaf >2014-04-07 12:25:53,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7afdb2d6 >2014-04-07 12:25:53,284 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 13:20:52,467 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-04-07 13:20:53,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-04-07 13:20:53,071 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VM OVFs in Data Center Lok >2014-04-07 13:20:53,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated VM OVFs in Data Center Lok >2014-04-07 13:20:53,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update template OVFs in Data Center Lok >2014-04-07 13:20:53,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated templates OVFs in Data Center Lok >2014-04-07 13:20:53,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 13:20:53,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 13:25:53,290 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 594b25e1 >2014-04-07 13:25:53,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 934c2f1 >2014-04-07 13:25:53,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 934c2f1 >2014-04-07 13:25:53,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 594b25e1 >2014-04-07 13:25:53,903 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 14:20:52,495 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-04-07 14:20:53,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-04-07 14:20:53,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VM OVFs in Data Center Lok >2014-04-07 14:20:53,081 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated VM OVFs in Data Center Lok >2014-04-07 14:20:53,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update template OVFs in Data Center Lok >2014-04-07 14:20:53,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated templates OVFs in Data Center Lok >2014-04-07 14:20:53,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 14:20:53,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 14:25:53,907 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2dd1f724 >2014-04-07 14:25:53,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 33b873fc >2014-04-07 14:25:54,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 33b873fc >2014-04-07 14:25:54,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2dd1f724 >2014-04-07 14:25:54,199 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 15:20:52,497 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-04-07 15:20:53,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-07 15:20:53,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-07 15:20:53,090 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-07 15:20:53,090 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-07 15:20:53,090 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-07 15:20:53,090 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 15:20:53,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 15:25:54,203 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 12c6643 >2014-04-07 15:25:54,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 9edb94f >2014-04-07 15:25:54,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 9edb94f >2014-04-07 15:25:54,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 12c6643 >2014-04-07 15:25:54,442 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 16:20:52,499 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-04-07 16:20:53,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-04-07 16:20:53,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VM OVFs in Data Center Lok >2014-04-07 16:20:53,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully updated VM OVFs in Data Center Lok >2014-04-07 16:20:53,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update template OVFs in Data Center Lok >2014-04-07 16:20:53,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully updated templates OVFs in Data Center Lok >2014-04-07 16:20:53,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 16:20:53,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 16:25:54,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: dacdd0b >2014-04-07 16:25:54,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4fe40430 >2014-04-07 16:25:54,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4fe40430 >2014-04-07 16:25:54,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: dacdd0b >2014-04-07 16:25:54,686 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 17:20:52,502 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-04-07 17:20:53,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-04-07 17:20:53,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VM OVFs in Data Center Lok >2014-04-07 17:20:53,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated VM OVFs in Data Center Lok >2014-04-07 17:20:53,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update template OVFs in Data Center Lok >2014-04-07 17:20:53,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated templates OVFs in Data Center Lok >2014-04-07 17:20:53,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 17:20:53,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 17:25:54,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 31cbae04 >2014-04-07 17:25:54,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 330da226 >2014-04-07 17:25:54,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 330da226 >2014-04-07 17:25:54,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 31cbae04 >2014-04-07 17:25:54,937 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 18:20:52,504 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-04-07 18:20:53,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-04-07 18:20:53,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VM OVFs in Data Center Lok >2014-04-07 18:20:53,100 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated VM OVFs in Data Center Lok >2014-04-07 18:20:53,100 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update template OVFs in Data Center Lok >2014-04-07 18:20:53,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated templates OVFs in Data Center Lok >2014-04-07 18:20:53,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 18:20:53,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 18:25:54,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 436eb5dc >2014-04-07 18:25:54,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1965a11a >2014-04-07 18:25:55,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1965a11a >2014-04-07 18:25:55,180 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 436eb5dc >2014-04-07 18:25:55,185 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 19:20:52,506 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-04-07 19:20:53,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-04-07 19:20:53,105 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VM OVFs in Data Center Lok >2014-04-07 19:20:53,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated VM OVFs in Data Center Lok >2014-04-07 19:20:53,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update template OVFs in Data Center Lok >2014-04-07 19:20:53,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated templates OVFs in Data Center Lok >2014-04-07 19:20:53,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 19:20:53,107 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 19:25:55,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 210e349f >2014-04-07 19:25:55,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2ba0c3ad >2014-04-07 19:25:55,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2ba0c3ad >2014-04-07 19:25:55,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 210e349f >2014-04-07 19:25:55,435 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 20:20:52,508 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-04-07 20:20:53,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-04-07 20:20:53,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VM OVFs in Data Center Lok >2014-04-07 20:20:53,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated VM OVFs in Data Center Lok >2014-04-07 20:20:53,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update template OVFs in Data Center Lok >2014-04-07 20:20:53,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated templates OVFs in Data Center Lok >2014-04-07 20:20:53,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 20:20:53,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 20:25:55,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2b9135ed >2014-04-07 20:25:55,439 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4a6b70f3 >2014-04-07 20:25:55,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4a6b70f3 >2014-04-07 20:25:55,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2b9135ed >2014-04-07 20:25:55,862 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 21:20:52,510 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-04-07 21:20:53,116 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-04-07 21:20:53,116 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VM OVFs in Data Center Lok >2014-04-07 21:20:53,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated VM OVFs in Data Center Lok >2014-04-07 21:20:53,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update template OVFs in Data Center Lok >2014-04-07 21:20:53,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated templates OVFs in Data Center Lok >2014-04-07 21:20:53,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 21:20:53,119 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 21:25:55,866 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 67b85477 >2014-04-07 21:25:55,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 76b185d3 >2014-04-07 21:25:56,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76b185d3 >2014-04-07 21:25:56,282 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 67b85477 >2014-04-07 21:25:56,293 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 22:20:52,510 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-04-07 22:20:53,119 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-04-07 22:20:53,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VM OVFs in Data Center Lok >2014-04-07 22:20:53,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated VM OVFs in Data Center Lok >2014-04-07 22:20:53,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update template OVFs in Data Center Lok >2014-04-07 22:20:53,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated templates OVFs in Data Center Lok >2014-04-07 22:20:53,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 22:20:53,122 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 22:25:56,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 65ff29df >2014-04-07 22:25:56,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 531623c9 >2014-04-07 22:25:56,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 531623c9 >2014-04-07 22:25:56,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 65ff29df >2014-04-07 22:25:56,545 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-07 23:20:52,512 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-04-07 23:20:53,122 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-04-07 23:20:53,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VM OVFs in Data Center Lok >2014-04-07 23:20:53,124 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated VM OVFs in Data Center Lok >2014-04-07 23:20:53,124 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update template OVFs in Data Center Lok >2014-04-07 23:20:53,124 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated templates OVFs in Data Center Lok >2014-04-07 23:20:53,124 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-07 23:20:53,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-07 23:25:56,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 18c669b4 >2014-04-07 23:25:56,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: a45b521 >2014-04-07 23:25:56,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: a45b521 >2014-04-07 23:25:56,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 18c669b4 >2014-04-07 23:25:56,798 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 00:20:52,514 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-04-08 00:20:53,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-08 00:20:53,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-08 00:20:53,127 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-08 00:20:53,127 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-08 00:20:53,128 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-08 00:20:53,128 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 00:20:53,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 00:25:56,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: cbb07b3 >2014-04-08 00:25:56,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1e495517 >2014-04-08 00:25:57,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1e495517 >2014-04-08 00:25:57,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: cbb07b3 >2014-04-08 00:25:57,052 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 01:20:52,516 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-04-08 01:20:53,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-04-08 01:20:53,132 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VM OVFs in Data Center Lok >2014-04-08 01:20:53,133 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated VM OVFs in Data Center Lok >2014-04-08 01:20:53,133 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update template OVFs in Data Center Lok >2014-04-08 01:20:53,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated templates OVFs in Data Center Lok >2014-04-08 01:20:53,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 01:20:53,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 01:25:57,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7a367c89 >2014-04-08 01:25:57,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: f2a450e >2014-04-08 01:25:57,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: f2a450e >2014-04-08 01:25:57,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7a367c89 >2014-04-08 01:25:57,303 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 02:20:52,516 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) Start refreshing all users data >2014-04-08 02:20:53,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-08 02:20:53,135 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-08 02:20:53,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-08 02:20:53,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-08 02:20:53,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-08 02:20:53,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 02:20:53,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 02:25:57,307 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 399c503d >2014-04-08 02:25:57,308 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4557f257 >2014-04-08 02:25:57,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4557f257 >2014-04-08 02:25:57,544 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 399c503d >2014-04-08 02:25:57,550 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 03:20:52,518 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-04-08 03:20:53,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-04-08 03:20:53,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VM OVFs in Data Center Lok >2014-04-08 03:20:53,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated VM OVFs in Data Center Lok >2014-04-08 03:20:53,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update template OVFs in Data Center Lok >2014-04-08 03:20:53,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated templates OVFs in Data Center Lok >2014-04-08 03:20:53,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 03:20:53,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 03:25:57,554 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 681463a >2014-04-08 03:25:57,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 11298d2c >2014-04-08 03:25:57,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 11298d2c >2014-04-08 03:25:57,791 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 681463a >2014-04-08 03:25:57,795 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-11) Start deleteAgedOutAuditLogs >2014-04-08 03:35:35,007 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-11) Finished deleteAgedOutAuditLogs >2014-04-08 04:20:52,519 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-08 04:20:53,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-04-08 04:20:53,141 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VM OVFs in Data Center Lok >2014-04-08 04:20:53,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated VM OVFs in Data Center Lok >2014-04-08 04:20:53,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update template OVFs in Data Center Lok >2014-04-08 04:20:53,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated templates OVFs in Data Center Lok >2014-04-08 04:20:53,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 04:20:53,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 04:25:57,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 34f3298b >2014-04-08 04:25:57,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 482c7fd5 >2014-04-08 04:25:58,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 482c7fd5 >2014-04-08 04:25:58,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 34f3298b >2014-04-08 04:25:58,053 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 05:20:52,521 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-04-08 05:20:53,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-04-08 05:20:53,144 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Lok >2014-04-08 05:20:53,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Lok >2014-04-08 05:20:53,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Lok >2014-04-08 05:20:53,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center Lok >2014-04-08 05:20:53,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 05:20:53,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 05:25:58,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6c0a5dc5 >2014-04-08 05:25:58,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1c9267ed >2014-04-08 05:25:58,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1c9267ed >2014-04-08 05:25:58,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6c0a5dc5 >2014-04-08 05:25:58,324 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 06:20:52,523 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-04-08 06:20:53,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-08 06:20:53,147 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-08 06:20:53,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-08 06:20:53,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-08 06:20:53,150 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-08 06:20:53,150 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 06:20:53,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 06:25:58,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: a52e1cc >2014-04-08 06:25:58,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 47661d5f >2014-04-08 06:25:58,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 47661d5f >2014-04-08 06:25:58,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: a52e1cc >2014-04-08 06:25:58,601 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 07:20:52,523 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-92) Start refreshing all users data >2014-04-08 07:20:53,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-08 07:20:53,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-08 07:20:53,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-08 07:20:53,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-08 07:20:53,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-08 07:20:53,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 07:20:53,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 07:25:58,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2b6f610e >2014-04-08 07:25:58,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1d4cf469 >2014-04-08 07:25:58,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d4cf469 >2014-04-08 07:25:58,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2b6f610e >2014-04-08 07:25:58,853 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 08:20:52,526 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-18) Start refreshing all users data >2014-04-08 08:20:53,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-04-08 08:20:53,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-04-08 08:20:53,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-04-08 08:20:53,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-04-08 08:20:53,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-04-08 08:20:53,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 08:20:53,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 08:25:58,856 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4e36baa9 >2014-04-08 08:25:58,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6b18d55c >2014-04-08 08:25:59,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6b18d55c >2014-04-08 08:25:59,083 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4e36baa9 >2014-04-08 08:25:59,088 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 09:20:52,527 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-04-08 09:20:53,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-04-08 09:20:53,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VM OVFs in Data Center Lok >2014-04-08 09:20:53,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated VM OVFs in Data Center Lok >2014-04-08 09:20:53,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update template OVFs in Data Center Lok >2014-04-08 09:20:53,163 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated templates OVFs in Data Center Lok >2014-04-08 09:20:53,163 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 09:20:53,164 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 09:25:59,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 607564a9 >2014-04-08 09:25:59,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3b23c4b8 >2014-04-08 09:25:59,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3b23c4b8 >2014-04-08 09:25:59,328 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 607564a9 >2014-04-08 09:25:59,333 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 10:20:52,527 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-04-08 10:20:53,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-04-08 10:20:53,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VM OVFs in Data Center Lok >2014-04-08 10:20:53,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated VM OVFs in Data Center Lok >2014-04-08 10:20:53,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update template OVFs in Data Center Lok >2014-04-08 10:20:53,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated templates OVFs in Data Center Lok >2014-04-08 10:20:53,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 10:20:53,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 10:25:59,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3531067a >2014-04-08 10:25:59,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 25a04ede >2014-04-08 10:25:59,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 25a04ede >2014-04-08 10:25:59,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3531067a >2014-04-08 10:25:59,582 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 11:20:52,529 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-04-08 11:20:53,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-04-08 11:20:53,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VM OVFs in Data Center Lok >2014-04-08 11:20:53,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated VM OVFs in Data Center Lok >2014-04-08 11:20:53,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update template OVFs in Data Center Lok >2014-04-08 11:20:53,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated templates OVFs in Data Center Lok >2014-04-08 11:20:53,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 11:20:53,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 11:25:59,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 23456f36 >2014-04-08 11:25:59,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 23885451 >2014-04-08 11:25:59,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23885451 >2014-04-08 11:25:59,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23456f36 >2014-04-08 11:25:59,849 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 12:20:52,529 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-08 12:20:53,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-08 12:20:53,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-04-08 12:20:53,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-04-08 12:20:53,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-04-08 12:20:53,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-04-08 12:20:53,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 12:20:53,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 12:25:59,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 64c2cf80 >2014-04-08 12:25:59,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 31c5102e >2014-04-08 12:26:00,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 31c5102e >2014-04-08 12:26:00,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 64c2cf80 >2014-04-08 12:26:00,093 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 13:20:52,530 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-89) Start refreshing all users data >2014-04-08 13:20:53,175 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VMs/Templates Ovf. >2014-04-08 13:20:53,176 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update VM OVFs in Data Center Lok >2014-04-08 13:20:53,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Successfully updated VM OVFs in Data Center Lok >2014-04-08 13:20:53,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to update template OVFs in Data Center Lok >2014-04-08 13:20:53,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Successfully updated templates OVFs in Data Center Lok >2014-04-08 13:20:53,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 13:20:53,178 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-79) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 13:26:00,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [4209317e] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: faa4e7d >2014-04-08 13:26:00,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [4209317e] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7606f893 >2014-04-08 13:26:00,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [4209317e] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7606f893 >2014-04-08 13:26:00,332 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [4209317e] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: faa4e7d >2014-04-08 13:26:00,341 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) [4209317e] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 14:20:52,532 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-04-08 14:20:53,179 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-04-08 14:20:53,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VM OVFs in Data Center Lok >2014-04-08 14:20:53,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated VM OVFs in Data Center Lok >2014-04-08 14:20:53,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update template OVFs in Data Center Lok >2014-04-08 14:20:53,181 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated templates OVFs in Data Center Lok >2014-04-08 14:20:53,181 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 14:20:53,181 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 14:26:00,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 203b186f >2014-04-08 14:26:00,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7c12ff1 >2014-04-08 14:26:00,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c12ff1 >2014-04-08 14:26:00,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 203b186f >2014-04-08 14:26:00,758 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 15:20:52,534 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-08 15:20:53,181 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-04-08 15:20:53,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VM OVFs in Data Center Lok >2014-04-08 15:20:53,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated VM OVFs in Data Center Lok >2014-04-08 15:20:53,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update template OVFs in Data Center Lok >2014-04-08 15:20:53,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated templates OVFs in Data Center Lok >2014-04-08 15:20:53,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 15:20:53,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 15:26:00,762 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1baa1ef2 >2014-04-08 15:26:00,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3dd5b00e >2014-04-08 15:26:01,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3dd5b00e >2014-04-08 15:26:01,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1baa1ef2 >2014-04-08 15:26:01,189 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 16:20:52,537 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-04-08 16:20:53,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-04-08 16:20:53,185 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VM OVFs in Data Center Lok >2014-04-08 16:20:53,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated VM OVFs in Data Center Lok >2014-04-08 16:20:53,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update template OVFs in Data Center Lok >2014-04-08 16:20:53,187 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated templates OVFs in Data Center Lok >2014-04-08 16:20:53,187 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 16:20:53,187 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 16:26:01,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7151ab1 >2014-04-08 16:26:01,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7a8b5e18 >2014-04-08 16:26:01,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7a8b5e18 >2014-04-08 16:26:01,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7151ab1 >2014-04-08 16:26:01,438 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 17:20:52,539 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-08 17:20:53,187 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-04-08 17:20:53,188 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VM OVFs in Data Center Lok >2014-04-08 17:20:53,188 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated VM OVFs in Data Center Lok >2014-04-08 17:20:53,188 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update template OVFs in Data Center Lok >2014-04-08 17:20:53,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated templates OVFs in Data Center Lok >2014-04-08 17:20:53,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 17:20:53,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 17:26:01,442 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1f736cb >2014-04-08 17:26:01,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: d2e4a6f >2014-04-08 17:26:01,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: d2e4a6f >2014-04-08 17:26:01,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1f736cb >2014-04-08 17:26:01,686 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 18:20:52,540 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-04-08 18:20:53,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-04-08 18:20:53,190 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VM OVFs in Data Center Lok >2014-04-08 18:20:53,191 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated VM OVFs in Data Center Lok >2014-04-08 18:20:53,191 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update template OVFs in Data Center Lok >2014-04-08 18:20:53,191 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated templates OVFs in Data Center Lok >2014-04-08 18:20:53,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 18:20:53,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 18:26:01,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7f489488 >2014-04-08 18:26:01,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 760fae00 >2014-04-08 18:26:01,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 760fae00 >2014-04-08 18:26:01,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f489488 >2014-04-08 18:26:01,939 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 19:20:52,542 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-08 19:20:53,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-04-08 19:20:53,193 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VM OVFs in Data Center Lok >2014-04-08 19:20:53,194 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated VM OVFs in Data Center Lok >2014-04-08 19:20:53,194 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update template OVFs in Data Center Lok >2014-04-08 19:20:53,194 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated templates OVFs in Data Center Lok >2014-04-08 19:20:53,194 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 19:20:53,195 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 19:26:01,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2634dd9e >2014-04-08 19:26:02,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 45c03c57 >2014-04-08 19:26:02,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 45c03c57 >2014-04-08 19:26:02,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2634dd9e >2014-04-08 19:26:02,705 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 20:20:52,544 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-04-08 20:20:53,195 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-04-08 20:20:53,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VM OVFs in Data Center Lok >2014-04-08 20:20:53,197 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully updated VM OVFs in Data Center Lok >2014-04-08 20:20:53,197 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update template OVFs in Data Center Lok >2014-04-08 20:20:53,197 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully updated templates OVFs in Data Center Lok >2014-04-08 20:20:53,197 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 20:20:53,198 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 20:26:02,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 64cb7184 >2014-04-08 20:26:02,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6889c4ce >2014-04-08 20:26:02,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6889c4ce >2014-04-08 20:26:02,943 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 64cb7184 >2014-04-08 20:26:02,947 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 21:20:52,546 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-04-08 21:20:53,198 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-04-08 21:20:53,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VM OVFs in Data Center Lok >2014-04-08 21:20:53,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated VM OVFs in Data Center Lok >2014-04-08 21:20:53,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update template OVFs in Data Center Lok >2014-04-08 21:20:53,200 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated templates OVFs in Data Center Lok >2014-04-08 21:20:53,200 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 21:20:53,200 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 21:26:02,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7c9ca9fc >2014-04-08 21:26:02,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 115c1f4e >2014-04-08 21:26:03,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 115c1f4e >2014-04-08 21:26:03,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c9ca9fc >2014-04-08 21:26:03,190 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 22:20:52,548 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-04-08 22:20:53,200 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-04-08 22:20:53,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VM OVFs in Data Center Lok >2014-04-08 22:20:53,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated VM OVFs in Data Center Lok >2014-04-08 22:20:53,202 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update template OVFs in Data Center Lok >2014-04-08 22:20:53,202 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated templates OVFs in Data Center Lok >2014-04-08 22:20:53,202 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 22:20:53,202 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 22:26:03,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7da5c69c >2014-04-08 22:26:03,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: e926c36 >2014-04-08 22:26:03,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: e926c36 >2014-04-08 22:26:03,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7da5c69c >2014-04-08 22:26:03,435 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-08 23:20:52,551 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-04-08 23:20:53,203 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-04-08 23:20:53,203 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VM OVFs in Data Center Lok >2014-04-08 23:20:53,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully updated VM OVFs in Data Center Lok >2014-04-08 23:20:53,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update template OVFs in Data Center Lok >2014-04-08 23:20:53,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully updated templates OVFs in Data Center Lok >2014-04-08 23:20:53,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-08 23:20:53,205 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-08 23:26:03,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 11d7b395 >2014-04-08 23:26:03,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 684f0459 >2014-04-08 23:26:03,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 684f0459 >2014-04-08 23:26:03,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 11d7b395 >2014-04-08 23:26:03,687 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 00:20:52,553 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-42) Start refreshing all users data >2014-04-09 00:20:53,205 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-04-09 00:20:53,206 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VM OVFs in Data Center Lok >2014-04-09 00:20:53,207 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated VM OVFs in Data Center Lok >2014-04-09 00:20:53,207 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update template OVFs in Data Center Lok >2014-04-09 00:20:53,207 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated templates OVFs in Data Center Lok >2014-04-09 00:20:53,207 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 00:20:53,209 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 00:26:03,692 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 54fc5291 >2014-04-09 00:26:03,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 190d92e >2014-04-09 00:26:03,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 190d92e >2014-04-09 00:26:03,932 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 54fc5291 >2014-04-09 00:26:03,937 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 01:20:52,555 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-04-09 01:20:53,210 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-04-09 01:20:53,210 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VM OVFs in Data Center Lok >2014-04-09 01:20:53,211 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Successfully updated VM OVFs in Data Center Lok >2014-04-09 01:20:53,211 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update template OVFs in Data Center Lok >2014-04-09 01:20:53,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Successfully updated templates OVFs in Data Center Lok >2014-04-09 01:20:53,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 01:20:53,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 01:26:03,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 11b2e1fe >2014-04-09 01:26:04,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 48f3f915 >2014-04-09 01:26:05,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 48f3f915 >2014-04-09 01:26:05,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 11b2e1fe >2014-04-09 01:26:05,111 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 02:20:52,557 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-09 02:20:53,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-04-09 02:20:53,213 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VM OVFs in Data Center Lok >2014-04-09 02:20:53,214 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated VM OVFs in Data Center Lok >2014-04-09 02:20:53,214 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update template OVFs in Data Center Lok >2014-04-09 02:20:53,215 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated templates OVFs in Data Center Lok >2014-04-09 02:20:53,215 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 02:20:53,215 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 02:26:05,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6934b3b0 >2014-04-09 02:26:05,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3011dc6 >2014-04-09 02:26:05,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3011dc6 >2014-04-09 02:26:05,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6934b3b0 >2014-04-09 02:26:05,356 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 03:20:52,559 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-66) Start refreshing all users data >2014-04-09 03:20:53,215 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-09 03:20:53,216 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-09 03:20:53,216 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-09 03:20:53,216 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-09 03:20:53,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-09 03:20:53,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 03:20:53,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 03:26:05,360 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 58c786af >2014-04-09 03:26:05,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 135b78b2 >2014-04-09 03:26:05,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 135b78b2 >2014-04-09 03:26:05,784 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 58c786af >2014-04-09 03:26:05,831 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-77) Start deleteAgedOutAuditLogs >2014-04-09 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-77) Finished deleteAgedOutAuditLogs >2014-04-09 04:20:52,563 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-04-09 04:20:53,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-04-09 04:20:53,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VM OVFs in Data Center Lok >2014-04-09 04:20:53,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated VM OVFs in Data Center Lok >2014-04-09 04:20:53,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update template OVFs in Data Center Lok >2014-04-09 04:20:53,219 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated templates OVFs in Data Center Lok >2014-04-09 04:20:53,219 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 04:20:53,219 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 04:26:05,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5e67d209 >2014-04-09 04:26:05,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 720e7d4a >2014-04-09 04:26:06,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 720e7d4a >2014-04-09 04:26:06,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e67d209 >2014-04-09 04:26:06,184 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 05:20:52,564 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-04-09 05:20:53,221 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-04-09 05:20:53,222 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VM OVFs in Data Center Lok >2014-04-09 05:20:53,223 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated VM OVFs in Data Center Lok >2014-04-09 05:20:53,223 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update template OVFs in Data Center Lok >2014-04-09 05:20:53,223 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated templates OVFs in Data Center Lok >2014-04-09 05:20:53,223 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 05:20:53,224 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 05:26:06,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2a6708e1 >2014-04-09 05:26:06,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 52780186 >2014-04-09 05:26:06,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 52780186 >2014-04-09 05:26:06,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2a6708e1 >2014-04-09 05:26:06,428 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 06:20:52,566 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-04-09 06:20:53,224 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-09 06:20:53,226 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-09 06:20:53,226 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-09 06:20:53,226 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-09 06:20:53,227 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-09 06:20:53,227 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 06:20:53,228 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 06:26:06,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5a3b09d5 >2014-04-09 06:26:06,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5a02ad3a >2014-04-09 06:26:06,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a02ad3a >2014-04-09 06:26:06,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a3b09d5 >2014-04-09 06:26:06,700 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 07:20:52,568 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-04-09 07:20:53,229 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-09 07:20:53,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-09 07:20:53,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-09 07:20:53,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-09 07:20:53,231 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-09 07:20:53,231 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 07:20:53,233 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 07:26:06,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2582acf0 >2014-04-09 07:26:06,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2627d3cd >2014-04-09 07:26:07,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2627d3cd >2014-04-09 07:26:07,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2582acf0 >2014-04-09 07:26:07,487 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 08:20:52,571 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-04-09 08:20:53,233 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-04-09 08:20:53,234 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VM OVFs in Data Center Lok >2014-04-09 08:20:53,239 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated VM OVFs in Data Center Lok >2014-04-09 08:20:53,239 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update template OVFs in Data Center Lok >2014-04-09 08:20:53,240 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated templates OVFs in Data Center Lok >2014-04-09 08:20:53,240 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 08:20:53,240 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 08:26:07,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5464bcd4 >2014-04-09 08:26:07,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 21a92d95 >2014-04-09 08:26:07,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 21a92d95 >2014-04-09 08:26:07,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5464bcd4 >2014-04-09 08:26:07,738 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 09:20:52,573 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-04-09 09:20:53,241 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-04-09 09:20:53,241 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VM OVFs in Data Center Lok >2014-04-09 09:20:53,242 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated VM OVFs in Data Center Lok >2014-04-09 09:20:53,242 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update template OVFs in Data Center Lok >2014-04-09 09:20:53,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated templates OVFs in Data Center Lok >2014-04-09 09:20:53,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 09:20:53,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 09:26:07,742 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 60bacfd9 >2014-04-09 09:26:07,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 8a3e12c >2014-04-09 09:26:08,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 8a3e12c >2014-04-09 09:26:08,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 60bacfd9 >2014-04-09 09:26:08,119 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 10:20:52,576 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-09 10:20:53,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-04-09 10:20:53,244 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VM OVFs in Data Center Lok >2014-04-09 10:20:53,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated VM OVFs in Data Center Lok >2014-04-09 10:20:53,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update template OVFs in Data Center Lok >2014-04-09 10:20:53,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated templates OVFs in Data Center Lok >2014-04-09 10:20:53,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 10:20:53,246 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 10:26:08,122 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2235df17 >2014-04-09 10:26:08,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7ec9708c >2014-04-09 10:26:08,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7ec9708c >2014-04-09 10:26:08,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2235df17 >2014-04-09 10:26:08,449 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 11:20:52,578 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) [4209317e] Start refreshing all users data >2014-04-09 11:20:53,246 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-04-09 11:20:53,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VM OVFs in Data Center Lok >2014-04-09 11:20:53,253 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated VM OVFs in Data Center Lok >2014-04-09 11:20:53,253 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update template OVFs in Data Center Lok >2014-04-09 11:20:53,258 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated templates OVFs in Data Center Lok >2014-04-09 11:20:53,258 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 11:20:53,258 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 11:26:08,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 30bb4588 >2014-04-09 11:26:08,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 56d1fcb6 >2014-04-09 11:26:08,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 56d1fcb6 >2014-04-09 11:26:08,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 30bb4588 >2014-04-09 11:26:08,701 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 12:20:52,580 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-09 12:20:53,258 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-04-09 12:20:53,258 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VM OVFs in Data Center Lok >2014-04-09 12:20:53,259 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated VM OVFs in Data Center Lok >2014-04-09 12:20:53,259 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update template OVFs in Data Center Lok >2014-04-09 12:20:53,260 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated templates OVFs in Data Center Lok >2014-04-09 12:20:53,260 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 12:20:53,260 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 12:26:08,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d7f569 >2014-04-09 12:26:08,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 930c171 >2014-04-09 12:26:08,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 930c171 >2014-04-09 12:26:08,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d7f569 >2014-04-09 12:26:08,950 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 13:20:52,581 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-04-09 13:20:53,260 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-04-09 13:20:53,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-04-09 13:20:53,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-04-09 13:20:53,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-04-09 13:20:53,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-04-09 13:20:53,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 13:20:53,263 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 13:26:08,954 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 353c0027 >2014-04-09 13:26:08,955 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 762e1bbe >2014-04-09 13:26:09,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 762e1bbe >2014-04-09 13:26:09,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 353c0027 >2014-04-09 13:26:09,201 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 14:20:52,582 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-87) Start refreshing all users data >2014-04-09 14:20:53,264 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-04-09 14:20:53,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Lok >2014-04-09 14:20:53,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Lok >2014-04-09 14:20:53,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Lok >2014-04-09 14:20:53,266 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated templates OVFs in Data Center Lok >2014-04-09 14:20:53,266 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 14:20:53,267 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 14:26:09,204 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 216c908f >2014-04-09 14:26:09,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5e86ad6d >2014-04-09 14:26:09,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e86ad6d >2014-04-09 14:26:09,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 216c908f >2014-04-09 14:26:09,437 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 15:20:52,584 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-04-09 15:20:53,267 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-09 15:20:53,268 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-09 15:20:53,269 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-09 15:20:53,269 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-09 15:20:53,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-09 15:20:53,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 15:20:53,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 15:26:09,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 233a703c >2014-04-09 15:26:09,444 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 519e8535 >2014-04-09 15:26:09,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 519e8535 >2014-04-09 15:26:09,871 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 233a703c >2014-04-09 15:26:09,876 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 16:20:52,586 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-04-09 16:20:53,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-04-09 16:20:53,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VM OVFs in Data Center Lok >2014-04-09 16:20:53,272 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated VM OVFs in Data Center Lok >2014-04-09 16:20:53,272 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update template OVFs in Data Center Lok >2014-04-09 16:20:53,272 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated templates OVFs in Data Center Lok >2014-04-09 16:20:53,273 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 16:20:53,273 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 16:26:09,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6f5f65e8 >2014-04-09 16:26:09,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 601b9b77 >2014-04-09 16:26:10,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 601b9b77 >2014-04-09 16:26:10,122 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6f5f65e8 >2014-04-09 16:26:10,128 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 17:20:52,588 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-04-09 17:20:53,273 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-04-09 17:20:53,273 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VM OVFs in Data Center Lok >2014-04-09 17:20:53,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated VM OVFs in Data Center Lok >2014-04-09 17:20:53,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update template OVFs in Data Center Lok >2014-04-09 17:20:53,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated templates OVFs in Data Center Lok >2014-04-09 17:20:53,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 17:20:53,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 17:26:10,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 30676f2d >2014-04-09 17:26:10,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 56b40c2e >2014-04-09 17:26:10,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 56b40c2e >2014-04-09 17:26:10,369 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 30676f2d >2014-04-09 17:26:10,373 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 18:20:52,591 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-04-09 18:20:53,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-04-09 18:20:53,276 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VM OVFs in Data Center Lok >2014-04-09 18:20:53,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated VM OVFs in Data Center Lok >2014-04-09 18:20:53,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update template OVFs in Data Center Lok >2014-04-09 18:20:53,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated templates OVFs in Data Center Lok >2014-04-09 18:20:53,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 18:20:53,278 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 18:26:10,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 155457c1 >2014-04-09 18:26:10,379 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 544d4ad7 >2014-04-09 18:26:10,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 544d4ad7 >2014-04-09 18:26:10,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 155457c1 >2014-04-09 18:26:10,621 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 19:20:52,591 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-09 19:20:53,278 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-04-09 19:20:53,279 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VM OVFs in Data Center Lok >2014-04-09 19:20:53,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated VM OVFs in Data Center Lok >2014-04-09 19:20:53,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update template OVFs in Data Center Lok >2014-04-09 19:20:53,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated templates OVFs in Data Center Lok >2014-04-09 19:20:53,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 19:20:53,281 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 19:26:10,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7c35dd68 >2014-04-09 19:26:10,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 345b168d >2014-04-09 19:26:10,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 345b168d >2014-04-09 19:26:10,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c35dd68 >2014-04-09 19:26:10,869 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 20:20:52,593 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-09 20:20:53,281 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-04-09 20:20:53,282 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VM OVFs in Data Center Lok >2014-04-09 20:20:53,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully updated VM OVFs in Data Center Lok >2014-04-09 20:20:53,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update template OVFs in Data Center Lok >2014-04-09 20:20:53,284 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully updated templates OVFs in Data Center Lok >2014-04-09 20:20:53,284 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 20:20:53,284 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 20:26:10,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7836a5cc >2014-04-09 20:26:10,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4bbf31bc >2014-04-09 20:26:11,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4bbf31bc >2014-04-09 20:26:11,121 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7836a5cc >2014-04-09 20:26:11,126 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 21:20:52,595 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-09 21:20:53,285 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-04-09 21:20:53,286 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Lok >2014-04-09 21:20:53,286 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Lok >2014-04-09 21:20:53,286 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Lok >2014-04-09 21:20:53,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center Lok >2014-04-09 21:20:53,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 21:20:53,288 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 21:26:11,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d5e32bd >2014-04-09 21:26:11,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1355255b >2014-04-09 21:26:11,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1355255b >2014-04-09 21:26:11,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d5e32bd >2014-04-09 21:26:11,371 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 22:20:52,597 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-04-09 22:20:53,288 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-04-09 22:20:53,289 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VM OVFs in Data Center Lok >2014-04-09 22:20:53,290 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated VM OVFs in Data Center Lok >2014-04-09 22:20:53,290 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update template OVFs in Data Center Lok >2014-04-09 22:20:53,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated templates OVFs in Data Center Lok >2014-04-09 22:20:53,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 22:20:53,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 22:26:11,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1956da0c >2014-04-09 22:26:11,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 150556d3 >2014-04-09 22:26:11,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 150556d3 >2014-04-09 22:26:11,614 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1956da0c >2014-04-09 22:26:11,619 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-09 23:20:52,599 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-66) Start refreshing all users data >2014-04-09 23:20:53,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-04-09 23:20:53,292 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VM OVFs in Data Center Lok >2014-04-09 23:20:53,292 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated VM OVFs in Data Center Lok >2014-04-09 23:20:53,292 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update template OVFs in Data Center Lok >2014-04-09 23:20:53,293 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated templates OVFs in Data Center Lok >2014-04-09 23:20:53,293 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-09 23:20:53,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-09 23:26:11,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 12e90b0d >2014-04-09 23:26:11,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1d48c2e >2014-04-09 23:26:11,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d48c2e >2014-04-09 23:26:11,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 12e90b0d >2014-04-09 23:26:11,868 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 00:20:52,601 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-10 00:20:53,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-04-10 00:20:53,295 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VM OVFs in Data Center Lok >2014-04-10 00:20:53,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated VM OVFs in Data Center Lok >2014-04-10 00:20:53,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update template OVFs in Data Center Lok >2014-04-10 00:20:53,298 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated templates OVFs in Data Center Lok >2014-04-10 00:20:53,298 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 00:20:53,298 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 00:26:11,872 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5f75ff68 >2014-04-10 00:26:11,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3bd335ce >2014-04-10 00:26:12,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3bd335ce >2014-04-10 00:26:12,106 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5f75ff68 >2014-04-10 00:26:12,110 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 01:20:52,603 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) [3fd33352] Start refreshing all users data >2014-04-10 01:20:53,298 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-04-10 01:20:53,302 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VM OVFs in Data Center Lok >2014-04-10 01:20:53,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated VM OVFs in Data Center Lok >2014-04-10 01:20:53,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update template OVFs in Data Center Lok >2014-04-10 01:20:53,304 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated templates OVFs in Data Center Lok >2014-04-10 01:20:53,304 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 01:20:53,304 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 01:26:12,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7f6d8d46 >2014-04-10 01:26:12,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 71563005 >2014-04-10 01:26:12,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 71563005 >2014-04-10 01:26:12,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f6d8d46 >2014-04-10 01:26:12,350 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 02:20:52,605 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-10 02:20:53,304 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-10 02:20:53,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-10 02:20:53,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-10 02:20:53,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-10 02:20:53,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-10 02:20:53,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 02:20:53,307 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 02:26:12,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 62e1aa0b >2014-04-10 02:26:12,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4d62eae4 >2014-04-10 02:26:12,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4d62eae4 >2014-04-10 02:26:12,584 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 62e1aa0b >2014-04-10 02:26:12,589 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 03:20:52,606 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-5) Start refreshing all users data >2014-04-10 03:20:53,307 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-04-10 03:20:53,308 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VM OVFs in Data Center Lok >2014-04-10 03:20:53,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated VM OVFs in Data Center Lok >2014-04-10 03:20:53,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update template OVFs in Data Center Lok >2014-04-10 03:20:53,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated templates OVFs in Data Center Lok >2014-04-10 03:20:53,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 03:20:53,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 03:26:12,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 357a6f7 >2014-04-10 03:26:12,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6f92a3d7 >2014-04-10 03:26:12,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6f92a3d7 >2014-04-10 03:26:12,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 357a6f7 >2014-04-10 03:26:13,000 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Start deleteAgedOutAuditLogs >2014-04-10 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-17) Finished deleteAgedOutAuditLogs >2014-04-10 04:20:52,609 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-04-10 04:20:53,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-04-10 04:20:53,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VM OVFs in Data Center Lok >2014-04-10 04:20:53,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated VM OVFs in Data Center Lok >2014-04-10 04:20:53,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update template OVFs in Data Center Lok >2014-04-10 04:20:53,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated templates OVFs in Data Center Lok >2014-04-10 04:20:53,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 04:20:53,314 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 04:26:13,006 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7c9e381a >2014-04-10 04:26:13,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 777572e0 >2014-04-10 04:26:13,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 777572e0 >2014-04-10 04:26:13,264 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c9e381a >2014-04-10 04:26:13,269 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 05:20:52,611 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-04-10 05:20:53,314 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-04-10 05:20:53,315 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VM OVFs in Data Center Lok >2014-04-10 05:20:53,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated VM OVFs in Data Center Lok >2014-04-10 05:20:53,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update template OVFs in Data Center Lok >2014-04-10 05:20:53,317 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated templates OVFs in Data Center Lok >2014-04-10 05:20:53,317 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 05:20:53,317 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 05:26:13,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 42d1f66f >2014-04-10 05:26:13,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 668a13ae >2014-04-10 05:26:13,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 668a13ae >2014-04-10 05:26:13,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 42d1f66f >2014-04-10 05:26:13,507 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 06:20:52,613 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-04-10 06:20:53,317 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-10 06:20:53,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-10 06:20:53,319 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-10 06:20:53,319 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-10 06:20:53,320 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-10 06:20:53,320 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 06:20:53,323 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 06:26:13,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3acfd7cc >2014-04-10 06:26:13,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 623729d4 >2014-04-10 06:26:13,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 623729d4 >2014-04-10 06:26:13,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3acfd7cc >2014-04-10 06:26:13,749 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 07:20:52,615 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-10 07:20:53,323 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-04-10 07:20:53,324 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VM OVFs in Data Center Lok >2014-04-10 07:20:53,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated VM OVFs in Data Center Lok >2014-04-10 07:20:53,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update template OVFs in Data Center Lok >2014-04-10 07:20:53,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated templates OVFs in Data Center Lok >2014-04-10 07:20:53,326 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 07:20:53,326 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 07:26:13,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 25b06767 >2014-04-10 07:26:13,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6054ae85 >2014-04-10 07:26:13,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6054ae85 >2014-04-10 07:26:13,982 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 25b06767 >2014-04-10 07:26:13,987 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 08:20:52,617 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-18) Start refreshing all users data >2014-04-10 08:20:53,326 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-10 08:20:53,327 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-10 08:20:53,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-10 08:20:53,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-10 08:20:53,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-10 08:20:53,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 08:20:53,329 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 08:26:13,991 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4e661e54 >2014-04-10 08:26:13,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 12514502 >2014-04-10 08:26:14,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 12514502 >2014-04-10 08:26:14,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4e661e54 >2014-04-10 08:26:14,405 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 09:20:52,619 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-04-10 09:20:53,329 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-10 09:20:53,330 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-10 09:20:53,330 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-10 09:20:53,330 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-10 09:20:53,331 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-10 09:20:53,331 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 09:20:53,332 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 09:26:14,408 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 37dbb782 >2014-04-10 09:26:14,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 39ecd474 >2014-04-10 09:26:14,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 39ecd474 >2014-04-10 09:26:14,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 37dbb782 >2014-04-10 09:26:14,652 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 10:20:52,621 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-04-10 10:20:53,332 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-04-10 10:20:53,333 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VM OVFs in Data Center Lok >2014-04-10 10:20:53,334 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated VM OVFs in Data Center Lok >2014-04-10 10:20:53,334 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update template OVFs in Data Center Lok >2014-04-10 10:20:53,335 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated templates OVFs in Data Center Lok >2014-04-10 10:20:53,335 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 10:20:53,335 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 10:26:14,656 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 470d203 >2014-04-10 10:26:14,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: aebc53c >2014-04-10 10:26:14,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: aebc53c >2014-04-10 10:26:14,890 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 470d203 >2014-04-10 10:26:14,894 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 11:20:52,623 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-04-10 11:20:53,336 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-04-10 11:20:53,337 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VM OVFs in Data Center Lok >2014-04-10 11:20:53,337 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated VM OVFs in Data Center Lok >2014-04-10 11:20:53,337 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update template OVFs in Data Center Lok >2014-04-10 11:20:53,338 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated templates OVFs in Data Center Lok >2014-04-10 11:20:53,338 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 11:20:53,339 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 11:26:14,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6aa29ced >2014-04-10 11:26:14,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 59ecc24c >2014-04-10 11:26:15,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 59ecc24c >2014-04-10 11:26:15,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6aa29ced >2014-04-10 11:26:15,301 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 12:20:52,625 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-04-10 12:20:53,340 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-10 12:20:53,341 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-04-10 12:20:53,341 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-04-10 12:20:53,341 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-04-10 12:20:53,342 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-04-10 12:20:53,342 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 12:20:53,342 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 12:26:15,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 121809d4 >2014-04-10 12:26:15,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 682d4693 >2014-04-10 12:26:15,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 682d4693 >2014-04-10 12:26:15,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 121809d4 >2014-04-10 12:26:15,538 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 13:20:52,626 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-04-10 13:20:53,342 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-04-10 13:20:53,343 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VM OVFs in Data Center Lok >2014-04-10 13:20:53,344 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated VM OVFs in Data Center Lok >2014-04-10 13:20:53,344 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update template OVFs in Data Center Lok >2014-04-10 13:20:53,344 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated templates OVFs in Data Center Lok >2014-04-10 13:20:53,345 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 13:20:53,345 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 13:26:15,542 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 164e7e27 >2014-04-10 13:26:15,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7b3c3de1 >2014-04-10 13:26:16,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7b3c3de1 >2014-04-10 13:26:16,174 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 164e7e27 >2014-04-10 13:26:16,180 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 14:20:52,628 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-04-10 14:20:53,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-04-10 14:20:53,347 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VM OVFs in Data Center Lok >2014-04-10 14:20:53,347 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated VM OVFs in Data Center Lok >2014-04-10 14:20:53,347 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update template OVFs in Data Center Lok >2014-04-10 14:20:53,348 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated templates OVFs in Data Center Lok >2014-04-10 14:20:53,348 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 14:20:53,348 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 14:26:16,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4f308a6a >2014-04-10 14:26:16,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7bcf5dc3 >2014-04-10 14:26:16,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7bcf5dc3 >2014-04-10 14:26:16,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f308a6a >2014-04-10 14:26:16,691 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 15:20:52,630 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) Start refreshing all users data >2014-04-10 15:20:53,349 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [3fd33352] Attempting to update VMs/Templates Ovf. >2014-04-10 15:20:53,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [3fd33352] Attempting to update VM OVFs in Data Center Lok >2014-04-10 15:20:53,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [3fd33352] Successfully updated VM OVFs in Data Center Lok >2014-04-10 15:20:53,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [3fd33352] Attempting to update template OVFs in Data Center Lok >2014-04-10 15:20:53,351 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [3fd33352] Successfully updated templates OVFs in Data Center Lok >2014-04-10 15:20:53,351 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [3fd33352] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 15:20:53,351 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [3fd33352] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 15:26:16,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d8b2455 >2014-04-10 15:26:16,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 19fe579a >2014-04-10 15:26:16,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 19fe579a >2014-04-10 15:26:16,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d8b2455 >2014-04-10 15:26:16,990 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 16:20:52,632 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-10 16:20:53,351 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-10 16:20:53,352 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-10 16:20:53,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-10 16:20:53,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-10 16:20:53,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-10 16:20:53,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 16:20:53,354 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 16:26:16,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2d23d619 >2014-04-10 16:26:16,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: bbd1a76 >2014-04-10 16:26:17,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: bbd1a76 >2014-04-10 16:26:17,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d23d619 >2014-04-10 16:26:17,231 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 17:20:52,634 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-04-10 17:20:53,354 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-04-10 17:20:53,356 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VM OVFs in Data Center Lok >2014-04-10 17:20:53,357 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully updated VM OVFs in Data Center Lok >2014-04-10 17:20:53,357 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update template OVFs in Data Center Lok >2014-04-10 17:20:53,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully updated templates OVFs in Data Center Lok >2014-04-10 17:20:53,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 17:20:53,359 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 17:26:17,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 114d8814 >2014-04-10 17:26:17,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7920e1eb >2014-04-10 17:26:17,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7920e1eb >2014-04-10 17:26:17,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 114d8814 >2014-04-10 17:26:17,470 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 18:20:52,634 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-6) Start refreshing all users data >2014-04-10 18:20:53,360 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-04-10 18:20:53,361 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VM OVFs in Data Center Lok >2014-04-10 18:20:53,361 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated VM OVFs in Data Center Lok >2014-04-10 18:20:53,361 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update template OVFs in Data Center Lok >2014-04-10 18:20:53,362 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated templates OVFs in Data Center Lok >2014-04-10 18:20:53,362 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 18:20:53,362 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 18:26:17,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6f81cdc6 >2014-04-10 18:26:17,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 14d6dda4 >2014-04-10 18:26:17,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 14d6dda4 >2014-04-10 18:26:17,818 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6f81cdc6 >2014-04-10 18:26:17,824 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 19:20:52,636 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) [4209317e] Start refreshing all users data >2014-04-10 19:20:53,362 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-04-10 19:20:53,363 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VM OVFs in Data Center Lok >2014-04-10 19:20:53,364 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully updated VM OVFs in Data Center Lok >2014-04-10 19:20:53,364 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update template OVFs in Data Center Lok >2014-04-10 19:20:53,364 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully updated templates OVFs in Data Center Lok >2014-04-10 19:20:53,364 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 19:20:53,366 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 19:26:17,827 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 173f4f6a >2014-04-10 19:26:17,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3d76c412 >2014-04-10 19:26:18,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3d76c412 >2014-04-10 19:26:18,060 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 173f4f6a >2014-04-10 19:26:18,064 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 20:20:52,638 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-04-10 20:20:53,366 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-04-10 20:20:53,367 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VM OVFs in Data Center Lok >2014-04-10 20:20:53,368 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully updated VM OVFs in Data Center Lok >2014-04-10 20:20:53,368 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update template OVFs in Data Center Lok >2014-04-10 20:20:53,368 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully updated templates OVFs in Data Center Lok >2014-04-10 20:20:53,368 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 20:20:53,369 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 20:26:18,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6a1f8318 >2014-04-10 20:26:18,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3ad7d316 >2014-04-10 20:26:18,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3ad7d316 >2014-04-10 20:26:18,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6a1f8318 >2014-04-10 20:26:18,627 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 21:20:52,639 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-04-10 21:20:53,369 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-04-10 21:20:53,370 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VM OVFs in Data Center Lok >2014-04-10 21:20:53,371 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated VM OVFs in Data Center Lok >2014-04-10 21:20:53,371 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update template OVFs in Data Center Lok >2014-04-10 21:20:53,371 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated templates OVFs in Data Center Lok >2014-04-10 21:20:53,371 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 21:20:53,372 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 21:26:18,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 56257520 >2014-04-10 21:26:18,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 352cb40f >2014-04-10 21:26:18,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 352cb40f >2014-04-10 21:26:18,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 56257520 >2014-04-10 21:26:18,864 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 22:20:52,641 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-66) Start refreshing all users data >2014-04-10 22:20:53,372 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-04-10 22:20:53,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VM OVFs in Data Center Lok >2014-04-10 22:20:53,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated VM OVFs in Data Center Lok >2014-04-10 22:20:53,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update template OVFs in Data Center Lok >2014-04-10 22:20:53,375 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated templates OVFs in Data Center Lok >2014-04-10 22:20:53,375 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 22:20:53,375 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 22:26:18,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d0a720e >2014-04-10 22:26:18,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 18c60407 >2014-04-10 22:26:19,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 18c60407 >2014-04-10 22:26:19,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d0a720e >2014-04-10 22:26:19,103 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-10 23:20:52,643 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-04-10 23:20:53,375 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-04-10 23:20:53,376 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VM OVFs in Data Center Lok >2014-04-10 23:20:53,378 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated VM OVFs in Data Center Lok >2014-04-10 23:20:53,378 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update template OVFs in Data Center Lok >2014-04-10 23:20:53,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated templates OVFs in Data Center Lok >2014-04-10 23:20:53,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-10 23:20:53,380 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-10 23:26:19,107 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1bd5b768 >2014-04-10 23:26:19,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6bfc1b1e >2014-04-10 23:26:19,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6bfc1b1e >2014-04-10 23:26:19,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1bd5b768 >2014-04-10 23:26:19,345 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 00:20:52,645 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-04-11 00:20:53,381 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-04-11 00:20:53,382 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VM OVFs in Data Center Lok >2014-04-11 00:20:53,382 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated VM OVFs in Data Center Lok >2014-04-11 00:20:53,382 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update template OVFs in Data Center Lok >2014-04-11 00:20:53,383 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated templates OVFs in Data Center Lok >2014-04-11 00:20:53,383 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 00:20:53,383 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 00:26:19,348 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4a7e255a >2014-04-11 00:26:19,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1fc65deb >2014-04-11 00:26:19,940 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1fc65deb >2014-04-11 00:26:19,940 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4a7e255a >2014-04-11 00:26:19,945 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 01:20:52,647 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-04-11 01:20:53,383 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-04-11 01:20:53,385 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VM OVFs in Data Center Lok >2014-04-11 01:20:53,385 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated VM OVFs in Data Center Lok >2014-04-11 01:20:53,385 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update template OVFs in Data Center Lok >2014-04-11 01:20:53,386 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated templates OVFs in Data Center Lok >2014-04-11 01:20:53,386 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 01:20:53,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 01:26:19,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 8821f5e >2014-04-11 01:26:19,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 16d958af >2014-04-11 01:26:20,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 16d958af >2014-04-11 01:26:20,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 8821f5e >2014-04-11 01:26:20,517 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 02:20:52,648 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-04-11 02:20:53,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-04-11 02:20:53,388 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VM OVFs in Data Center Lok >2014-04-11 02:20:53,389 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated VM OVFs in Data Center Lok >2014-04-11 02:20:53,389 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update template OVFs in Data Center Lok >2014-04-11 02:20:53,390 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated templates OVFs in Data Center Lok >2014-04-11 02:20:53,390 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 02:20:53,391 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 02:26:20,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 33fe00d5 >2014-04-11 02:26:20,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 68e8ba9d >2014-04-11 02:26:20,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 68e8ba9d >2014-04-11 02:26:20,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 33fe00d5 >2014-04-11 02:26:20,753 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 03:20:52,650 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-92) Start refreshing all users data >2014-04-11 03:20:53,392 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-11 03:20:53,392 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-11 03:20:53,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-11 03:20:53,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-11 03:20:53,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-11 03:20:53,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 03:20:53,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 03:26:20,756 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4d13f31b >2014-04-11 03:26:20,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 58bd9925 >2014-04-11 03:26:20,987 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 58bd9925 >2014-04-11 03:26:20,987 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4d13f31b >2014-04-11 03:26:20,992 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) Start deleteAgedOutAuditLogs >2014-04-11 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) Finished deleteAgedOutAuditLogs >2014-04-11 04:20:52,652 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-11 04:20:53,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-11 04:20:53,395 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-11 04:20:53,396 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-11 04:20:53,396 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-11 04:20:53,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-11 04:20:53,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 04:20:53,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 04:26:20,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5767a527 >2014-04-11 04:26:20,996 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 558bd045 >2014-04-11 04:26:21,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 558bd045 >2014-04-11 04:26:21,222 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5767a527 >2014-04-11 04:26:21,227 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 05:20:52,654 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-86) Start refreshing all users data >2014-04-11 05:20:53,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-11 05:20:53,398 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-11 05:20:53,398 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-11 05:20:53,399 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-11 05:20:53,399 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-11 05:20:53,399 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 05:20:53,400 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 05:26:21,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7e2cc195 >2014-04-11 05:26:21,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2c595dea >2014-04-11 05:26:21,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2c595dea >2014-04-11 05:26:21,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7e2cc195 >2014-04-11 05:26:21,470 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 06:20:52,656 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-04-11 06:20:53,400 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-04-11 06:20:53,401 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VM OVFs in Data Center Lok >2014-04-11 06:20:53,402 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated VM OVFs in Data Center Lok >2014-04-11 06:20:53,402 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update template OVFs in Data Center Lok >2014-04-11 06:20:53,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated templates OVFs in Data Center Lok >2014-04-11 06:20:53,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 06:20:53,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 06:26:21,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 61d45b14 >2014-04-11 06:26:21,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3ca7b039 >2014-04-11 06:26:21,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3ca7b039 >2014-04-11 06:26:21,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 61d45b14 >2014-04-11 06:26:21,710 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 06:50:25,670 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginAdminUserCommand internal: false. >2014-04-11 06:50:25,677 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-11 06:50:27,003 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-04-11 06:50:27,011 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-11 06:52:08,877 INFO [org.ovirt.engine.core.bll.UpdateVmCommand] (ajp-/127.0.0.1:8702-3) [65592ad4] Lock Acquired to object EngineLock [exclusiveLocks= key: Gal value: VM_NAME >, sharedLocks= ] >2014-04-11 06:52:08,963 INFO [org.ovirt.engine.core.bll.UpdateVmCommand] (ajp-/127.0.0.1:8702-3) [65592ad4] Running command: UpdateVmCommand internal: false. Entities affected : ID: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 Type: VM, ID: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 Type: VM >2014-04-11 06:52:09,042 INFO [org.ovirt.engine.core.bll.AddWatchdogCommand] (ajp-/127.0.0.1:8702-3) [5a98fe89] Running command: AddWatchdogCommand internal: true. Entities affected : ID: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 Type: VM >2014-04-11 06:52:09,063 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [5a98fe89] Correlation ID: 65592ad4, Job ID: 47da44d5-836c-48f5-b871-4efad302d92b, Call Stack: null, Custom Event ID: -1, Message: VM Gal configuration was updated by admin@internal. >2014-04-11 06:52:09,079 INFO [org.ovirt.engine.core.bll.UpdateVmCommand] (ajp-/127.0.0.1:8702-3) [5a98fe89] Lock freed to object EngineLock [exclusiveLocks= key: Gal value: VM_NAME >, sharedLocks= ] >2014-04-11 06:52:09,796 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (ajp-/127.0.0.1:8702-20) MultipleActionsRunner of type AddVmInterface invoked with no actions >2014-04-11 06:52:10,823 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (ajp-/127.0.0.1:8702-21) MultipleActionsRunner of type UpdateVmInterface invoked with no actions >2014-04-11 06:52:11,176 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (ajp-/127.0.0.1:8702-4) MultipleActionsRunner of type RemoveVmInterface invoked with no actions >2014-04-11 06:52:47,356 INFO [org.ovirt.engine.core.bll.RunVmCommand] (ajp-/127.0.0.1:8702-20) [5620f607] Lock Acquired to object EngineLock [exclusiveLocks= key: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 value: VM >, sharedLocks= ] >2014-04-11 06:52:47,431 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-20) [5620f607] START, IsVmDuringInitiatingVDSCommand( vmId = b066ff4b-970f-4305-94a4-1f32f7fa6fe2), log id: 3b66a7cf >2014-04-11 06:52:47,431 INFO [org.ovirt.engine.core.vdsbroker.IsVmDuringInitiatingVDSCommand] (ajp-/127.0.0.1:8702-20) [5620f607] FINISH, IsVmDuringInitiatingVDSCommand, return: false, log id: 3b66a7cf >2014-04-11 06:52:47,483 INFO [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-48) [5620f607] Running command: RunVmCommand internal: false. Entities affected : ID: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 Type: VM >2014-04-11 06:52:47,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoDirectoryVDSCommand] (pool-4-thread-48) [5620f607] START, IsoDirectoryVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4e6c72ee >2014-04-11 06:52:47,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IsoDirectoryVDSCommand] (pool-4-thread-48) [5620f607] FINISH, IsoDirectoryVDSCommand, return: \\10.34.27.209\CD, log id: 4e6c72ee >2014-04-11 06:52:47,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (pool-4-thread-48) [5620f607] START, IsoPrefixVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2966f183 >2014-04-11 06:52:47,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.IsoPrefixVDSCommand] (pool-4-thread-48) [5620f607] FINISH, IsoPrefixVDSCommand, return: /rhev/data-center/mnt/10.34.27.209:_mnt_iso/6ba7421b-4b80-4aff-ba01-1c08d3ef4619/images/11111111-1111-1111-1111-111111111111, log id: 2966f183 >2014-04-11 06:52:47,948 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-48) [5620f607] START, CreateVmVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, vm=VM [Gal]), log id: 708060c0 >2014-04-11 06:52:47,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-48) [5620f607] START, CreateVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, vm=VM [Gal]), log id: 6b2c7131 >2014-04-11 06:52:48,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-48) [5620f607] org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand spiceSslCipherSuite=DEFAULT,memSize=1024,kvmEnable=true,smp=1,vmType=kvm,emulatedMachine=rhel6.4.0,keyboardLayout=en-us,memGuaranteedSize=1024,nice=0,display=qxl,smartcardEnable=false,smpCoresPerSocket=1,spiceSecureChannels=smain,sinputs,scursor,splayback,srecord,sdisplay,susbredir,ssmartcard,timeOffset=-2,transparentHugePages=true,vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2,devices=[Ljava.util.HashMap;@8d5764f,acpiEnable=true,vmName=Gal,cpuType=Conroe,custom={device_4ab025d9-b0b0-40b9-8b19-523280b74c36device_8fe884df-d429-42bf-bd8b-e2f15fe703eadevice_aa243a33-315b-447b-8673-e4389f6de2cedevice_cfc0123b-72d1-4547-aae3-240ec5c91936=VmDevice {vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, deviceId=cfc0123b-72d1-4547-aae3-240ec5c91936, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=2, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel1, customProperties={}, snapshotId=null}, qemu_cmdline=["-global","qxl-vga.debug=5","-global","qxl-vga.guestdebug=1"], device_4ab025d9-b0b0-40b9-8b19-523280b74c36device_8fe884df-d429-42bf-bd8b-e2f15fe703eadevice_aa243a33-315b-447b-8673-e4389f6de2ce=VmDevice {vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, deviceId=aa243a33-315b-447b-8673-e4389f6de2ce, device=spicevmc, type=CHANNEL, bootOrder=0, specParams={}, address={port=3, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel2, customProperties={}, snapshotId=null}, device_4ab025d9-b0b0-40b9-8b19-523280b74c36=VmDevice {vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, deviceId=4ab025d9-b0b0-40b9-8b19-523280b74c36, device=unix, type=CHANNEL, bootOrder=0, specParams={}, address={port=1, bus=0, controller=0, type=virtio-serial}, managed=false, plugged=true, readOnly=false, deviceAlias=channel0, customProperties={}, snapshotId=null}, device_4ab025d9-b0b0-40b9-8b19-523280b74c36device_8fe884df-d429-42bf-bd8b-e2f15fe703eadevice_aa243a33-315b-447b-8673-e4389f6de2cedevice_cfc0123b-72d1-4547-aae3-240ec5c91936device_f4abc7f7-ce1e-4d01-bc2a-1a4c688d8eb1=VmDevice {vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, deviceId=f4abc7f7-ce1e-4d01-bc2a-1a4c688d8eb1, device=ide, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x01, function=0x1}, managed=false, plugged=true, readOnly=false, deviceAlias=ide0, customProperties={}, snapshotId=null}, device_4ab025d9-b0b0-40b9-8b19-523280b74c36device_8fe884df-d429-42bf-bd8b-e2f15fe703ea=VmDevice {vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, deviceId=8fe884df-d429-42bf-bd8b-e2f15fe703ea, device=virtio-serial, type=CONTROLLER, bootOrder=0, specParams={}, address={bus=0x00, domain=0x0000, type=pci, slot=0x05, function=0x0}, managed=false, plugged=true, readOnly=false, deviceAlias=virtio-serial0, customProperties={}, snapshotId=null}, device_4ab025d9-b0b0-40b9-8b19-523280b74c36device_8fe884df-d429-42bf-bd8b-e2f15fe703eadevice_aa243a33-315b-447b-8673-e4389f6de2cedevice_cfc0123b-72d1-4547-aae3-240ec5c91936device_f4abc7f7-ce1e-4d01-bc2a-1a4c688d8eb1device_00000000-0000-0000-0000-000000000000=VmDevice {vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, deviceId=00000000-0000-0000-0000-000000000000, device=watchdog, type=WATCHDOG, bootOrder=0, specParams={model=i6300esb, action=reset}, address=, managed=false, plugged=false, readOnly=false, deviceAlias=, customProperties={}, snapshotId=null}} >2014-04-11 06:52:48,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateVDSCommand] (pool-4-thread-48) [5620f607] FINISH, CreateVDSCommand, log id: 6b2c7131 >2014-04-11 06:52:48,298 INFO [org.ovirt.engine.core.vdsbroker.CreateVmVDSCommand] (pool-4-thread-48) [5620f607] FINISH, CreateVmVDSCommand, return: WaitForLaunch, log id: 708060c0 >2014-04-11 06:52:48,298 INFO [org.ovirt.engine.core.bll.RunVmCommand] (pool-4-thread-48) [5620f607] Lock freed to object EngineLock [exclusiveLocks= key: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 value: VM >, sharedLocks= ] >2014-04-11 06:52:48,308 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-48) [5620f607] Correlation ID: 5620f607, Job ID: 7a42d400-3db7-423e-b3de-41a08591b309, Call Stack: null, Custom Event ID: -1, Message: VM Gal was started by admin@internal (Host: 10.34.27.209). >2014-04-11 06:52:49,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-4) START, FullListVdsCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209], vmIds=[b066ff4b-970f-4305-94a4-1f32f7fa6fe2]), log id: 1cd5efa6 >2014-04-11 06:52:49,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-4) FINISH, FullListVdsCommand, return: [Ljava.util.HashMap;@21688a14, log id: 1cd5efa6 >2014-04-11 06:52:52,949 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) VM Gal b066ff4b-970f-4305-94a4-1f32f7fa6fe2 moved from WaitForLaunch --> PoweringUp >2014-04-11 06:52:52,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-74) START, FullListVdsCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209], vmIds=[b066ff4b-970f-4305-94a4-1f32f7fa6fe2]), log id: 594f7f3e >2014-04-11 06:52:53,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FullListVdsCommand] (DefaultQuartzScheduler_Worker-74) FINISH, FullListVdsCommand, return: [Ljava.util.HashMap;@603e3f4e, log id: 594f7f3e >2014-04-11 06:53:48,564 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) VM Gal b066ff4b-970f-4305-94a4-1f32f7fa6fe2 moved from PoweringUp --> Up >2014-04-11 06:53:48,572 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) Correlation ID: 5620f607, Job ID: 7a42d400-3db7-423e-b3de-41a08591b309, Call Stack: null, Custom Event ID: -1, Message: VM Gal started on Host 10.34.27.209 >2014-04-11 07:20:52,658 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-11 07:20:53,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-04-11 07:20:53,405 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VM OVFs in Data Center Lok >2014-04-11 07:20:53,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (DefaultQuartzScheduler_Worker-54) START, UpdateVMVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 2aa005d5 >2014-04-11 07:20:53,880 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, UpdateVMVDSCommand, log id: 2aa005d5 >2014-04-11 07:20:53,885 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated VM OVFs in Data Center Lok >2014-04-11 07:20:53,885 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update template OVFs in Data Center Lok >2014-04-11 07:20:53,886 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated templates OVFs in Data Center Lok >2014-04-11 07:20:53,886 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 07:20:53,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 07:26:21,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 580fdca7 >2014-04-11 07:26:21,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 795f851b >2014-04-11 07:26:21,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 795f851b >2014-04-11 07:26:21,948 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 580fdca7 >2014-04-11 07:26:21,953 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 08:20:52,659 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-04-11 08:20:53,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-04-11 08:20:53,889 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VM OVFs in Data Center Lok >2014-04-11 08:20:53,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated VM OVFs in Data Center Lok >2014-04-11 08:20:53,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update template OVFs in Data Center Lok >2014-04-11 08:20:53,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated templates OVFs in Data Center Lok >2014-04-11 08:20:53,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 08:20:53,897 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 08:26:21,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5de661a6 >2014-04-11 08:26:21,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 40150973 >2014-04-11 08:26:22,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 40150973 >2014-04-11 08:26:22,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5de661a6 >2014-04-11 08:26:22,195 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 09:20:52,662 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-11 09:20:53,897 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-04-11 09:20:53,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VM OVFs in Data Center Lok >2014-04-11 09:20:53,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated VM OVFs in Data Center Lok >2014-04-11 09:20:53,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update template OVFs in Data Center Lok >2014-04-11 09:20:53,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated templates OVFs in Data Center Lok >2014-04-11 09:20:53,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 09:20:53,905 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 09:26:22,198 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 545c0eb1 >2014-04-11 09:26:22,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 67c8b1bb >2014-04-11 09:26:22,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 67c8b1bb >2014-04-11 09:26:22,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 545c0eb1 >2014-04-11 09:26:22,440 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 10:20:52,664 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-04-11 10:20:53,905 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-04-11 10:20:53,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VM OVFs in Data Center Lok >2014-04-11 10:20:53,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated VM OVFs in Data Center Lok >2014-04-11 10:20:53,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update template OVFs in Data Center Lok >2014-04-11 10:20:53,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated templates OVFs in Data Center Lok >2014-04-11 10:20:53,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 10:20:53,913 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 10:26:22,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 29da2a08 >2014-04-11 10:26:22,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 31fd88af >2014-04-11 10:26:22,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 31fd88af >2014-04-11 10:26:22,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 29da2a08 >2014-04-11 10:26:22,685 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 11:20:52,667 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-04-11 11:20:53,913 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-04-11 11:20:53,914 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VM OVFs in Data Center Lok >2014-04-11 11:20:53,915 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated VM OVFs in Data Center Lok >2014-04-11 11:20:53,915 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update template OVFs in Data Center Lok >2014-04-11 11:20:53,915 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated templates OVFs in Data Center Lok >2014-04-11 11:20:53,915 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 11:20:53,916 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 11:26:22,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 492efaf >2014-04-11 11:26:22,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4ab5cc81 >2014-04-11 11:26:22,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4ab5cc81 >2014-04-11 11:26:22,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 492efaf >2014-04-11 11:26:22,922 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 12:20:52,670 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-04-11 12:20:53,917 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-04-11 12:20:53,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VM OVFs in Data Center Lok >2014-04-11 12:20:53,919 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated VM OVFs in Data Center Lok >2014-04-11 12:20:53,919 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update template OVFs in Data Center Lok >2014-04-11 12:20:53,919 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated templates OVFs in Data Center Lok >2014-04-11 12:20:53,919 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 12:20:53,920 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 12:26:22,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 45c6fd00 >2014-04-11 12:26:22,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: b0c2eca >2014-04-11 12:26:23,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: b0c2eca >2014-04-11 12:26:23,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 45c6fd00 >2014-04-11 12:26:23,160 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 13:20:52,672 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-04-11 13:20:53,920 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-04-11 13:20:53,921 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-04-11 13:20:53,922 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-04-11 13:20:53,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-04-11 13:20:53,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-04-11 13:20:53,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 13:20:53,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 13:26:23,163 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 736c41de >2014-04-11 13:26:23,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 48589105 >2014-04-11 13:26:23,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 48589105 >2014-04-11 13:26:23,422 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 736c41de >2014-04-11 13:26:23,426 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 14:20:52,674 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-04-11 14:20:53,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-04-11 14:20:53,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VM OVFs in Data Center Lok >2014-04-11 14:20:53,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated VM OVFs in Data Center Lok >2014-04-11 14:20:53,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update template OVFs in Data Center Lok >2014-04-11 14:20:53,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated templates OVFs in Data Center Lok >2014-04-11 14:20:53,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 14:20:53,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 14:26:23,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: a0ab85b >2014-04-11 14:26:23,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 45cd6383 >2014-04-11 14:26:23,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 45cd6383 >2014-04-11 14:26:23,905 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: a0ab85b >2014-04-11 14:26:23,909 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 15:20:52,676 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-04-11 15:20:53,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-04-11 15:20:53,929 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VM OVFs in Data Center Lok >2014-04-11 15:20:53,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated VM OVFs in Data Center Lok >2014-04-11 15:20:53,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update template OVFs in Data Center Lok >2014-04-11 15:20:53,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated templates OVFs in Data Center Lok >2014-04-11 15:20:53,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 15:20:53,931 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 15:26:23,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7905982b >2014-04-11 15:26:23,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5e496d1f >2014-04-11 15:26:24,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e496d1f >2014-04-11 15:26:24,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7905982b >2014-04-11 15:26:24,153 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 16:20:52,678 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-04-11 16:20:53,931 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-04-11 16:20:53,932 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-04-11 16:20:53,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-04-11 16:20:53,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-04-11 16:20:53,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-04-11 16:20:53,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 16:20:53,934 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 16:26:24,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5e140cf7 >2014-04-11 16:26:24,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 765f5c61 >2014-04-11 16:26:24,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 765f5c61 >2014-04-11 16:26:24,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e140cf7 >2014-04-11 16:26:24,388 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 17:20:52,680 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-04-11 17:20:53,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-04-11 17:20:53,936 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VM OVFs in Data Center Lok >2014-04-11 17:20:53,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Successfully updated VM OVFs in Data Center Lok >2014-04-11 17:20:53,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update template OVFs in Data Center Lok >2014-04-11 17:20:53,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Successfully updated templates OVFs in Data Center Lok >2014-04-11 17:20:53,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 17:20:53,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 17:26:24,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3b4ce63e >2014-04-11 17:26:24,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 35a00edc >2014-04-11 17:26:24,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 35a00edc >2014-04-11 17:26:24,619 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3b4ce63e >2014-04-11 17:26:24,623 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 18:20:52,682 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-04-11 18:20:53,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-04-11 18:20:53,939 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-04-11 18:20:53,940 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-04-11 18:20:53,940 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-04-11 18:20:53,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-04-11 18:20:53,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 18:20:53,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 18:26:24,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6c38d047 >2014-04-11 18:26:24,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5f21cd3 >2014-04-11 18:26:24,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5f21cd3 >2014-04-11 18:26:24,857 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6c38d047 >2014-04-11 18:26:24,862 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 19:20:52,683 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-11 19:20:53,942 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-04-11 19:20:53,942 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VM OVFs in Data Center Lok >2014-04-11 19:20:53,943 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated VM OVFs in Data Center Lok >2014-04-11 19:20:53,943 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update template OVFs in Data Center Lok >2014-04-11 19:20:53,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated templates OVFs in Data Center Lok >2014-04-11 19:20:53,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 19:20:53,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 19:26:24,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2a518ae2 >2014-04-11 19:26:24,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4f751122 >2014-04-11 19:26:25,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f751122 >2014-04-11 19:26:25,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2a518ae2 >2014-04-11 19:26:25,096 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 20:20:52,685 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-04-11 20:20:53,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-11 20:20:53,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-11 20:20:53,946 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-11 20:20:53,946 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-11 20:20:53,947 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-11 20:20:53,947 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 20:20:53,948 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 20:26:25,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4f4436b4 >2014-04-11 20:26:25,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7638c14 >2014-04-11 20:26:25,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7638c14 >2014-04-11 20:26:25,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f4436b4 >2014-04-11 20:26:25,506 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 21:20:52,688 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-04-11 21:20:53,948 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-04-11 21:20:53,949 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VM OVFs in Data Center Lok >2014-04-11 21:20:53,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Successfully updated VM OVFs in Data Center Lok >2014-04-11 21:20:53,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update template OVFs in Data Center Lok >2014-04-11 21:20:53,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Successfully updated templates OVFs in Data Center Lok >2014-04-11 21:20:53,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 21:20:53,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 21:26:25,508 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1b1b551f >2014-04-11 21:26:25,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1f717420 >2014-04-11 21:26:25,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1f717420 >2014-04-11 21:26:25,937 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b1b551f >2014-04-11 21:26:25,941 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 22:20:52,689 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-04-11 22:20:53,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-04-11 22:20:53,952 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VM OVFs in Data Center Lok >2014-04-11 22:20:53,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated VM OVFs in Data Center Lok >2014-04-11 22:20:53,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update template OVFs in Data Center Lok >2014-04-11 22:20:53,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated templates OVFs in Data Center Lok >2014-04-11 22:20:53,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 22:20:53,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 22:26:25,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 56c9a231 >2014-04-11 22:26:25,945 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4fb76b7a >2014-04-11 22:26:26,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4fb76b7a >2014-04-11 22:26:26,169 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 56c9a231 >2014-04-11 22:26:26,174 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-11 23:20:52,691 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-04-11 23:20:53,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-04-11 23:20:53,955 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VM OVFs in Data Center Lok >2014-04-11 23:20:53,956 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated VM OVFs in Data Center Lok >2014-04-11 23:20:53,956 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update template OVFs in Data Center Lok >2014-04-11 23:20:53,956 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated templates OVFs in Data Center Lok >2014-04-11 23:20:53,956 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-11 23:20:53,957 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-11 23:26:26,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 36ca3b54 >2014-04-11 23:26:26,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 16fd23e6 >2014-04-11 23:26:26,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 16fd23e6 >2014-04-11 23:26:26,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 36ca3b54 >2014-04-11 23:26:26,431 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 00:20:52,692 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-04-12 00:20:53,957 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-12 00:20:53,958 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-04-12 00:20:53,959 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-04-12 00:20:53,959 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-04-12 00:20:53,959 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-04-12 00:20:53,959 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 00:20:53,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 00:26:26,434 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7d714bb >2014-04-12 00:26:26,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 39920502 >2014-04-12 00:26:26,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 39920502 >2014-04-12 00:26:26,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7d714bb >2014-04-12 00:26:26,665 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 01:20:52,694 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-12 01:20:53,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-04-12 01:20:53,961 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VM OVFs in Data Center Lok >2014-04-12 01:20:53,962 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated VM OVFs in Data Center Lok >2014-04-12 01:20:53,962 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update template OVFs in Data Center Lok >2014-04-12 01:20:53,962 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated templates OVFs in Data Center Lok >2014-04-12 01:20:53,962 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 01:20:53,963 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 01:26:26,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 321e5abb >2014-04-12 01:26:26,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 66d6c4b1 >2014-04-12 01:26:26,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 66d6c4b1 >2014-04-12 01:26:26,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 321e5abb >2014-04-12 01:26:26,898 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 02:20:52,696 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-42) Start refreshing all users data >2014-04-12 02:20:53,963 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-04-12 02:20:53,964 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VM OVFs in Data Center Lok >2014-04-12 02:20:53,965 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated VM OVFs in Data Center Lok >2014-04-12 02:20:53,965 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update template OVFs in Data Center Lok >2014-04-12 02:20:53,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated templates OVFs in Data Center Lok >2014-04-12 02:20:53,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 02:20:53,967 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 02:26:26,903 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1a72aacf >2014-04-12 02:26:26,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5e2c0999 >2014-04-12 02:26:27,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e2c0999 >2014-04-12 02:26:27,130 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1a72aacf >2014-04-12 02:26:27,135 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 03:20:52,698 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-04-12 03:20:53,967 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-12 03:20:53,968 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-12 03:20:53,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-12 03:20:53,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-12 03:20:53,970 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-12 03:20:53,970 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 03:20:53,971 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 03:26:27,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7b0dc48c >2014-04-12 03:26:27,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 44e30271 >2014-04-12 03:26:27,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 44e30271 >2014-04-12 03:26:27,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7b0dc48c >2014-04-12 03:26:27,402 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-74) Start deleteAgedOutAuditLogs >2014-04-12 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-74) Finished deleteAgedOutAuditLogs >2014-04-12 04:20:52,700 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-04-12 04:20:53,971 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-04-12 04:20:53,972 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-04-12 04:20:53,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-04-12 04:20:53,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-04-12 04:20:53,974 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-04-12 04:20:53,974 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 04:20:53,974 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 04:26:27,407 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2fb94fe0 >2014-04-12 04:26:27,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2f2b139a >2014-04-12 04:26:27,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2f2b139a >2014-04-12 04:26:27,635 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2fb94fe0 >2014-04-12 04:26:27,640 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 05:13:06,722 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) VM Gal b066ff4b-970f-4305-94a4-1f32f7fa6fe2 moved from Up --> RebootInProgress >2014-04-12 05:14:01,752 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) VM Gal b066ff4b-970f-4305-94a4-1f32f7fa6fe2 moved from RebootInProgress --> Up >2014-04-12 05:20:52,702 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-12 05:20:53,975 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-04-12 05:20:53,976 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VM OVFs in Data Center Lok >2014-04-12 05:20:53,977 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated VM OVFs in Data Center Lok >2014-04-12 05:20:53,977 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update template OVFs in Data Center Lok >2014-04-12 05:20:53,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated templates OVFs in Data Center Lok >2014-04-12 05:20:53,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 05:20:53,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 05:26:27,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 772c9677 >2014-04-12 05:26:27,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 104a9bc1 >2014-04-12 05:26:27,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 104a9bc1 >2014-04-12 05:26:27,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 772c9677 >2014-04-12 05:26:27,874 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 06:20:52,703 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-20) Start refreshing all users data >2014-04-12 06:20:53,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-04-12 06:20:53,979 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VM OVFs in Data Center Lok >2014-04-12 06:20:53,980 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated VM OVFs in Data Center Lok >2014-04-12 06:20:53,980 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update template OVFs in Data Center Lok >2014-04-12 06:20:53,980 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated templates OVFs in Data Center Lok >2014-04-12 06:20:53,980 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 06:20:53,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 06:26:27,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 532b17b5 >2014-04-12 06:26:27,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 322196bb >2014-04-12 06:26:28,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 322196bb >2014-04-12 06:26:28,102 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 532b17b5 >2014-04-12 06:26:28,107 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 07:20:52,705 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-04-12 07:20:53,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-04-12 07:20:53,983 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VM OVFs in Data Center Lok >2014-04-12 07:20:53,983 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated VM OVFs in Data Center Lok >2014-04-12 07:20:53,983 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update template OVFs in Data Center Lok >2014-04-12 07:20:53,984 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated templates OVFs in Data Center Lok >2014-04-12 07:20:53,984 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 07:20:53,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 07:26:28,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5defa2bd >2014-04-12 07:26:28,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1fa55679 >2014-04-12 07:26:28,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1fa55679 >2014-04-12 07:26:28,337 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5defa2bd >2014-04-12 07:26:28,342 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 08:20:52,707 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-04-12 08:20:53,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-04-12 08:20:53,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VM OVFs in Data Center Lok >2014-04-12 08:20:53,986 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated VM OVFs in Data Center Lok >2014-04-12 08:20:53,986 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update template OVFs in Data Center Lok >2014-04-12 08:20:53,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated templates OVFs in Data Center Lok >2014-04-12 08:20:53,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 08:20:53,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 08:26:28,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2269721d >2014-04-12 08:26:28,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 38c2df91 >2014-04-12 08:26:28,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 38c2df91 >2014-04-12 08:26:28,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2269721d >2014-04-12 08:26:28,571 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 09:20:52,709 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-04-12 09:20:53,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-04-12 09:20:53,988 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VM OVFs in Data Center Lok >2014-04-12 09:20:53,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated VM OVFs in Data Center Lok >2014-04-12 09:20:53,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update template OVFs in Data Center Lok >2014-04-12 09:20:53,990 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated templates OVFs in Data Center Lok >2014-04-12 09:20:53,990 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 09:20:53,991 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 09:26:28,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 13421d4a >2014-04-12 09:26:28,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5990ef26 >2014-04-12 09:26:28,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5990ef26 >2014-04-12 09:26:28,794 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 13421d4a >2014-04-12 09:26:28,799 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 10:20:52,711 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-04-12 10:20:53,991 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-04-12 10:20:53,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VM OVFs in Data Center Lok >2014-04-12 10:20:53,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated VM OVFs in Data Center Lok >2014-04-12 10:20:53,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update template OVFs in Data Center Lok >2014-04-12 10:20:53,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated templates OVFs in Data Center Lok >2014-04-12 10:20:53,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 10:20:53,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 10:26:28,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 76e88d9c >2014-04-12 10:26:28,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1da939c1 >2014-04-12 10:26:29,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1da939c1 >2014-04-12 10:26:29,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76e88d9c >2014-04-12 10:26:29,030 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 11:20:52,713 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-04-12 11:20:53,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-04-12 11:20:53,996 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VM OVFs in Data Center Lok >2014-04-12 11:20:53,997 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated VM OVFs in Data Center Lok >2014-04-12 11:20:53,997 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update template OVFs in Data Center Lok >2014-04-12 11:20:53,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated templates OVFs in Data Center Lok >2014-04-12 11:20:53,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 11:20:53,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 11:26:29,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 784cbd7c >2014-04-12 11:26:29,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 762c5597 >2014-04-12 11:26:29,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 762c5597 >2014-04-12 11:26:29,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 784cbd7c >2014-04-12 11:26:29,264 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 12:20:52,715 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-12 12:20:53,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-04-12 12:20:53,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VM OVFs in Data Center Lok >2014-04-12 12:20:54,000 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Successfully updated VM OVFs in Data Center Lok >2014-04-12 12:20:54,000 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update template OVFs in Data Center Lok >2014-04-12 12:20:54,000 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Successfully updated templates OVFs in Data Center Lok >2014-04-12 12:20:54,000 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 12:20:54,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 12:26:29,269 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 449cb85b >2014-04-12 12:26:29,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: d0b818e >2014-04-12 12:26:29,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: d0b818e >2014-04-12 12:26:29,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 449cb85b >2014-04-12 12:26:29,498 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 13:20:52,716 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-20) Start refreshing all users data >2014-04-12 13:20:54,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-04-12 13:20:54,003 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-04-12 13:20:54,004 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-04-12 13:20:54,004 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-04-12 13:20:54,005 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-04-12 13:20:54,005 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 13:20:54,005 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 13:26:29,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2b3c2bba >2014-04-12 13:26:29,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7b5ad682 >2014-04-12 13:26:29,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7b5ad682 >2014-04-12 13:26:29,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2b3c2bba >2014-04-12 13:26:29,736 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 14:20:52,718 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-62) Start refreshing all users data >2014-04-12 14:20:54,005 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-04-12 14:20:54,006 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VM OVFs in Data Center Lok >2014-04-12 14:20:54,007 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated VM OVFs in Data Center Lok >2014-04-12 14:20:54,007 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update template OVFs in Data Center Lok >2014-04-12 14:20:54,007 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated templates OVFs in Data Center Lok >2014-04-12 14:20:54,007 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 14:20:54,008 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 14:26:29,740 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 80eeef0 >2014-04-12 14:26:29,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: c68d2b2 >2014-04-12 14:26:29,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: c68d2b2 >2014-04-12 14:26:29,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 80eeef0 >2014-04-12 14:26:29,967 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 15:20:52,721 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-5) Start refreshing all users data >2014-04-12 15:20:54,008 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-04-12 15:20:54,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VM OVFs in Data Center Lok >2014-04-12 15:20:54,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated VM OVFs in Data Center Lok >2014-04-12 15:20:54,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update template OVFs in Data Center Lok >2014-04-12 15:20:54,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated templates OVFs in Data Center Lok >2014-04-12 15:20:54,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 15:20:54,011 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 15:26:29,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 563a8bcb >2014-04-12 15:26:29,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1dafe8a7 >2014-04-12 15:26:30,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1dafe8a7 >2014-04-12 15:26:30,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 563a8bcb >2014-04-12 15:26:30,223 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 16:20:52,723 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-42) Start refreshing all users data >2014-04-12 16:20:54,011 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-04-12 16:20:54,012 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Lok >2014-04-12 16:20:54,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Lok >2014-04-12 16:20:54,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Lok >2014-04-12 16:20:54,014 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated templates OVFs in Data Center Lok >2014-04-12 16:20:54,014 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 16:20:54,015 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 16:26:30,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 44141ead >2014-04-12 16:26:30,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4ea12c6 >2014-04-12 16:26:30,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4ea12c6 >2014-04-12 16:26:30,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 44141ead >2014-04-12 16:26:30,455 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 17:20:52,724 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-12 17:20:54,022 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-12 17:20:54,023 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-12 17:20:54,024 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-12 17:20:54,024 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-12 17:20:54,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-12 17:20:54,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 17:20:54,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 17:26:30,459 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3c22be5c >2014-04-12 17:26:30,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: cffa8dc >2014-04-12 17:26:30,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: cffa8dc >2014-04-12 17:26:30,684 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c22be5c >2014-04-12 17:26:30,690 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 18:20:52,726 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-57) Start refreshing all users data >2014-04-12 18:20:54,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-04-12 18:20:54,026 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VM OVFs in Data Center Lok >2014-04-12 18:20:54,027 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated VM OVFs in Data Center Lok >2014-04-12 18:20:54,027 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update template OVFs in Data Center Lok >2014-04-12 18:20:54,028 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated templates OVFs in Data Center Lok >2014-04-12 18:20:54,028 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 18:20:54,028 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 18:26:30,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4f3502b3 >2014-04-12 18:26:30,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 21445e65 >2014-04-12 18:26:31,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 21445e65 >2014-04-12 18:26:31,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f3502b3 >2014-04-12 18:26:31,086 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 19:20:52,728 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-04-12 19:20:54,028 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-04-12 19:20:54,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VM OVFs in Data Center Lok >2014-04-12 19:20:54,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated VM OVFs in Data Center Lok >2014-04-12 19:20:54,030 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update template OVFs in Data Center Lok >2014-04-12 19:20:54,030 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated templates OVFs in Data Center Lok >2014-04-12 19:20:54,030 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 19:20:54,031 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 19:26:31,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5f29768a >2014-04-12 19:26:31,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1b258ed5 >2014-04-12 19:26:31,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b258ed5 >2014-04-12 19:26:31,313 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5f29768a >2014-04-12 19:26:31,318 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 20:20:52,730 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-87) Start refreshing all users data >2014-04-12 20:20:54,031 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-04-12 20:20:54,032 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-04-12 20:20:54,032 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-04-12 20:20:54,032 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-04-12 20:20:54,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-04-12 20:20:54,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 20:20:54,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 20:26:31,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1c37d889 >2014-04-12 20:26:31,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 166918d8 >2014-04-12 20:26:31,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 166918d8 >2014-04-12 20:26:31,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1c37d889 >2014-04-12 20:26:31,552 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 21:20:52,731 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-62) Start refreshing all users data >2014-04-12 21:20:54,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-04-12 21:20:54,034 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VM OVFs in Data Center Lok >2014-04-12 21:20:54,035 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated VM OVFs in Data Center Lok >2014-04-12 21:20:54,035 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update template OVFs in Data Center Lok >2014-04-12 21:20:54,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated templates OVFs in Data Center Lok >2014-04-12 21:20:54,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 21:20:54,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 21:26:31,557 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 42406f17 >2014-04-12 21:26:31,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 143d70fc >2014-04-12 21:26:31,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 143d70fc >2014-04-12 21:26:31,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 42406f17 >2014-04-12 21:26:31,806 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 22:20:52,733 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-12 22:20:54,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-04-12 22:20:54,037 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VM OVFs in Data Center Lok >2014-04-12 22:20:54,038 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated VM OVFs in Data Center Lok >2014-04-12 22:20:54,038 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update template OVFs in Data Center Lok >2014-04-12 22:20:54,038 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated templates OVFs in Data Center Lok >2014-04-12 22:20:54,038 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 22:20:54,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 22:26:31,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 58907935 >2014-04-12 22:26:31,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7505b3ff >2014-04-12 22:26:32,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7505b3ff >2014-04-12 22:26:32,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 58907935 >2014-04-12 22:26:32,040 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-12 23:20:52,735 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-69) Start refreshing all users data >2014-04-12 23:20:54,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-04-12 23:20:54,040 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Lok >2014-04-12 23:20:54,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Lok >2014-04-12 23:20:54,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Lok >2014-04-12 23:20:54,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center Lok >2014-04-12 23:20:54,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-12 23:20:54,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-12 23:26:32,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5cb35cc >2014-04-12 23:26:32,044 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6038e70c >2014-04-12 23:26:32,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6038e70c >2014-04-12 23:26:32,268 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5cb35cc >2014-04-12 23:26:32,272 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 00:20:52,736 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-04-13 00:20:54,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-04-13 00:20:54,043 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VM OVFs in Data Center Lok >2014-04-13 00:20:54,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated VM OVFs in Data Center Lok >2014-04-13 00:20:54,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update template OVFs in Data Center Lok >2014-04-13 00:20:54,045 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated templates OVFs in Data Center Lok >2014-04-13 00:20:54,045 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 00:20:54,045 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 00:26:32,276 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7a0aff0f >2014-04-13 00:26:32,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 67ef7061 >2014-04-13 00:26:32,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 67ef7061 >2014-04-13 00:26:32,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7a0aff0f >2014-04-13 00:26:32,510 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 01:20:52,738 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-04-13 01:20:54,046 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-04-13 01:20:54,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VM OVFs in Data Center Lok >2014-04-13 01:20:54,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated VM OVFs in Data Center Lok >2014-04-13 01:20:54,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update template OVFs in Data Center Lok >2014-04-13 01:20:54,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated templates OVFs in Data Center Lok >2014-04-13 01:20:54,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 01:20:54,049 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 01:26:32,514 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 8665004 >2014-04-13 01:26:32,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 750f3aea >2014-04-13 01:26:32,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 750f3aea >2014-04-13 01:26:32,735 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 8665004 >2014-04-13 01:26:32,740 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 02:20:52,741 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-04-13 02:20:54,050 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-04-13 02:20:54,051 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Lok >2014-04-13 02:20:54,051 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Lok >2014-04-13 02:20:54,051 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Lok >2014-04-13 02:20:54,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated templates OVFs in Data Center Lok >2014-04-13 02:20:54,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 02:20:54,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 02:26:32,744 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4da4b95d >2014-04-13 02:26:32,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7985b07d >2014-04-13 02:26:32,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7985b07d >2014-04-13 02:26:32,970 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4da4b95d >2014-04-13 02:26:32,975 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 03:20:52,743 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-04-13 03:20:54,053 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-04-13 03:20:54,054 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VM OVFs in Data Center Lok >2014-04-13 03:20:54,054 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated VM OVFs in Data Center Lok >2014-04-13 03:20:54,054 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update template OVFs in Data Center Lok >2014-04-13 03:20:54,055 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated templates OVFs in Data Center Lok >2014-04-13 03:20:54,055 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 03:20:54,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 03:26:32,980 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6d7d4a3c >2014-04-13 03:26:32,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3c94e14c >2014-04-13 03:26:33,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c94e14c >2014-04-13 03:26:33,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6d7d4a3c >2014-04-13 03:26:33,207 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-79) Start deleteAgedOutAuditLogs >2014-04-13 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-79) Finished deleteAgedOutAuditLogs >2014-04-13 04:20:52,745 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-04-13 04:20:54,057 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-04-13 04:20:54,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VM OVFs in Data Center Lok >2014-04-13 04:20:54,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated VM OVFs in Data Center Lok >2014-04-13 04:20:54,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update template OVFs in Data Center Lok >2014-04-13 04:20:54,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated templates OVFs in Data Center Lok >2014-04-13 04:20:54,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 04:20:54,060 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 04:26:33,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 45dfc827 >2014-04-13 04:26:33,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 66cd9c47 >2014-04-13 04:26:33,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 66cd9c47 >2014-04-13 04:26:33,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 45dfc827 >2014-04-13 04:26:33,441 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 05:20:52,747 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-04-13 05:20:54,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-04-13 05:20:54,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VM OVFs in Data Center Lok >2014-04-13 05:20:54,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated VM OVFs in Data Center Lok >2014-04-13 05:20:54,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update template OVFs in Data Center Lok >2014-04-13 05:20:54,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated templates OVFs in Data Center Lok >2014-04-13 05:20:54,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 05:20:54,064 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 05:26:33,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 615ce59c >2014-04-13 05:26:33,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 19e696b4 >2014-04-13 05:26:33,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 19e696b4 >2014-04-13 05:26:33,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 615ce59c >2014-04-13 05:26:33,671 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 06:20:52,747 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-04-13 06:20:54,064 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-04-13 06:20:54,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VM OVFs in Data Center Lok >2014-04-13 06:20:54,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated VM OVFs in Data Center Lok >2014-04-13 06:20:54,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update template OVFs in Data Center Lok >2014-04-13 06:20:54,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated templates OVFs in Data Center Lok >2014-04-13 06:20:54,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 06:20:54,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 06:26:33,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6fecccf4 >2014-04-13 06:26:33,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2383e094 >2014-04-13 06:26:33,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2383e094 >2014-04-13 06:26:33,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6fecccf4 >2014-04-13 06:26:33,904 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 07:20:52,750 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-13 07:20:54,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-04-13 07:20:54,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VM OVFs in Data Center Lok >2014-04-13 07:20:54,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated VM OVFs in Data Center Lok >2014-04-13 07:20:54,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update template OVFs in Data Center Lok >2014-04-13 07:20:54,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated templates OVFs in Data Center Lok >2014-04-13 07:20:54,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 07:20:54,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 07:26:33,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6290281d >2014-04-13 07:26:33,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6639ba78 >2014-04-13 07:26:34,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6639ba78 >2014-04-13 07:26:34,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6290281d >2014-04-13 07:26:34,137 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 08:20:52,752 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-81) Start refreshing all users data >2014-04-13 08:20:54,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-04-13 08:20:54,071 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VM OVFs in Data Center Lok >2014-04-13 08:20:54,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated VM OVFs in Data Center Lok >2014-04-13 08:20:54,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update template OVFs in Data Center Lok >2014-04-13 08:20:54,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated templates OVFs in Data Center Lok >2014-04-13 08:20:54,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 08:20:54,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 08:26:34,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7592a16a >2014-04-13 08:26:34,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6da37273 >2014-04-13 08:26:34,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6da37273 >2014-04-13 08:26:34,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7592a16a >2014-04-13 08:26:34,367 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 09:20:52,754 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-62) Start refreshing all users data >2014-04-13 09:20:54,074 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-04-13 09:20:54,074 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-04-13 09:20:54,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-04-13 09:20:54,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-04-13 09:20:54,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-04-13 09:20:54,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 09:20:54,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 09:26:34,373 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4dfa2c52 >2014-04-13 09:26:34,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6373e37c >2014-04-13 09:26:34,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6373e37c >2014-04-13 09:26:34,596 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4dfa2c52 >2014-04-13 09:26:34,601 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 10:20:52,755 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-04-13 10:20:54,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-04-13 10:20:54,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VM OVFs in Data Center Lok >2014-04-13 10:20:54,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully updated VM OVFs in Data Center Lok >2014-04-13 10:20:54,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update template OVFs in Data Center Lok >2014-04-13 10:20:54,079 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully updated templates OVFs in Data Center Lok >2014-04-13 10:20:54,079 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 10:20:54,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 10:26:34,604 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 20094659 >2014-04-13 10:26:34,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 40e98d7d >2014-04-13 10:26:34,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 40e98d7d >2014-04-13 10:26:34,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 20094659 >2014-04-13 10:26:34,834 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 11:20:52,757 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-13 11:20:54,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-13 11:20:54,081 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-13 11:20:54,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-13 11:20:54,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-13 11:20:54,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-13 11:20:54,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 11:20:54,083 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 11:26:34,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 371765c >2014-04-13 11:26:34,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: e1193c8 >2014-04-13 11:26:35,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: e1193c8 >2014-04-13 11:26:35,063 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 371765c >2014-04-13 11:26:35,068 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 12:20:52,759 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-04-13 12:20:54,083 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-04-13 12:20:54,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VM OVFs in Data Center Lok >2014-04-13 12:20:54,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated VM OVFs in Data Center Lok >2014-04-13 12:20:54,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update template OVFs in Data Center Lok >2014-04-13 12:20:54,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated templates OVFs in Data Center Lok >2014-04-13 12:20:54,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 12:20:54,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 12:26:35,073 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 38ff097e >2014-04-13 12:26:35,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6fc84c6d >2014-04-13 12:26:35,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6fc84c6d >2014-04-13 12:26:35,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 38ff097e >2014-04-13 12:26:35,358 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 13:20:52,761 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-04-13 13:20:54,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-04-13 13:20:54,087 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Lok >2014-04-13 13:20:54,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Lok >2014-04-13 13:20:54,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Lok >2014-04-13 13:20:54,089 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated templates OVFs in Data Center Lok >2014-04-13 13:20:54,089 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 13:20:54,089 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 13:26:35,362 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6cb874bc >2014-04-13 13:26:35,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 41f81a67 >2014-04-13 13:26:35,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 41f81a67 >2014-04-13 13:26:35,606 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6cb874bc >2014-04-13 13:26:35,612 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 14:20:52,763 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) Start refreshing all users data >2014-04-13 14:20:54,090 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-04-13 14:20:54,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VM OVFs in Data Center Lok >2014-04-13 14:20:54,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated VM OVFs in Data Center Lok >2014-04-13 14:20:54,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update template OVFs in Data Center Lok >2014-04-13 14:20:54,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated templates OVFs in Data Center Lok >2014-04-13 14:20:54,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 14:20:54,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 14:26:35,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 299dba58 >2014-04-13 14:26:35,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3234fcc0 >2014-04-13 14:26:35,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3234fcc0 >2014-04-13 14:26:35,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 299dba58 >2014-04-13 14:26:35,861 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 15:20:52,765 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-13 15:20:54,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-04-13 15:20:54,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VM OVFs in Data Center Lok >2014-04-13 15:20:54,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated VM OVFs in Data Center Lok >2014-04-13 15:20:54,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update template OVFs in Data Center Lok >2014-04-13 15:20:54,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated templates OVFs in Data Center Lok >2014-04-13 15:20:54,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 15:20:54,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 15:26:35,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1fa72d24 >2014-04-13 15:26:35,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1884a6b8 >2014-04-13 15:26:36,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1884a6b8 >2014-04-13 15:26:36,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1fa72d24 >2014-04-13 15:26:36,099 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 16:20:52,767 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-04-13 16:20:54,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VMs/Templates Ovf. >2014-04-13 16:20:54,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update VM OVFs in Data Center Lok >2014-04-13 16:20:54,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Successfully updated VM OVFs in Data Center Lok >2014-04-13 16:20:54,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to update template OVFs in Data Center Lok >2014-04-13 16:20:54,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Successfully updated templates OVFs in Data Center Lok >2014-04-13 16:20:54,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 16:20:54,100 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-4) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 16:26:36,104 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 189fb0f3 >2014-04-13 16:26:36,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1c1a2e26 >2014-04-13 16:26:36,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1c1a2e26 >2014-04-13 16:26:36,325 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 189fb0f3 >2014-04-13 16:26:36,330 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 17:20:52,769 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-04-13 17:20:54,100 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-13 17:20:54,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-04-13 17:20:54,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-04-13 17:20:54,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-04-13 17:20:54,103 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-04-13 17:20:54,103 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 17:20:54,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 17:26:36,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 57040c53 >2014-04-13 17:26:36,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4ac549c6 >2014-04-13 17:26:36,555 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4ac549c6 >2014-04-13 17:26:36,556 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 57040c53 >2014-04-13 17:26:36,560 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 18:20:52,771 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-04-13 18:20:54,105 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VMs/Templates Ovf. >2014-04-13 18:20:54,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update VM OVFs in Data Center Lok >2014-04-13 18:20:54,107 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Successfully updated VM OVFs in Data Center Lok >2014-04-13 18:20:54,107 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to update template OVFs in Data Center Lok >2014-04-13 18:20:54,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Successfully updated templates OVFs in Data Center Lok >2014-04-13 18:20:54,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 18:20:54,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-45) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 18:26:36,563 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2020e67e >2014-04-13 18:26:36,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4ffc6aac >2014-04-13 18:26:36,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4ffc6aac >2014-04-13 18:26:36,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2020e67e >2014-04-13 18:26:36,795 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 19:20:52,773 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-04-13 19:20:54,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-04-13 19:20:54,109 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VM OVFs in Data Center Lok >2014-04-13 19:20:54,110 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated VM OVFs in Data Center Lok >2014-04-13 19:20:54,110 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update template OVFs in Data Center Lok >2014-04-13 19:20:54,110 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated templates OVFs in Data Center Lok >2014-04-13 19:20:54,110 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 19:20:54,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 19:26:36,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1e5f96fe >2014-04-13 19:26:36,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 15b20d46 >2014-04-13 19:26:37,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 15b20d46 >2014-04-13 19:26:37,028 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1e5f96fe >2014-04-13 19:26:37,032 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 20:20:52,775 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-04-13 20:20:54,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-04-13 20:20:54,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-04-13 20:20:54,113 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-04-13 20:20:54,113 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-04-13 20:20:54,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-04-13 20:20:54,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 20:20:54,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 20:26:37,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5e402710 >2014-04-13 20:26:37,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 45c4bbc7 >2014-04-13 20:26:37,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 45c4bbc7 >2014-04-13 20:26:37,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e402710 >2014-04-13 20:26:37,264 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 21:20:52,776 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-04-13 21:20:54,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-04-13 21:20:54,116 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VM OVFs in Data Center Lok >2014-04-13 21:20:54,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated VM OVFs in Data Center Lok >2014-04-13 21:20:54,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update template OVFs in Data Center Lok >2014-04-13 21:20:54,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated templates OVFs in Data Center Lok >2014-04-13 21:20:54,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 21:20:54,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 21:26:37,267 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2e9a1fec >2014-04-13 21:26:37,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 8a5d6db >2014-04-13 21:26:37,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 8a5d6db >2014-04-13 21:26:37,496 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e9a1fec >2014-04-13 21:26:37,506 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 22:20:52,778 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-92) Start refreshing all users data >2014-04-13 22:20:54,119 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-04-13 22:20:54,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VM OVFs in Data Center Lok >2014-04-13 22:20:54,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully updated VM OVFs in Data Center Lok >2014-04-13 22:20:54,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update template OVFs in Data Center Lok >2014-04-13 22:20:54,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully updated templates OVFs in Data Center Lok >2014-04-13 22:20:54,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 22:20:54,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 22:26:37,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5857d9af >2014-04-13 22:26:37,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6f949d78 >2014-04-13 22:26:38,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6f949d78 >2014-04-13 22:26:38,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5857d9af >2014-04-13 22:26:38,007 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-13 23:20:52,780 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-04-13 23:20:54,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-04-13 23:20:54,122 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VM OVFs in Data Center Lok >2014-04-13 23:20:54,122 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated VM OVFs in Data Center Lok >2014-04-13 23:20:54,122 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update template OVFs in Data Center Lok >2014-04-13 23:20:54,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated templates OVFs in Data Center Lok >2014-04-13 23:20:54,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-13 23:20:54,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-13 23:26:38,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 44c97906 >2014-04-13 23:26:38,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7289c32 >2014-04-13 23:26:38,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7289c32 >2014-04-13 23:26:38,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 44c97906 >2014-04-13 23:26:38,240 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 00:20:52,782 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-14 00:20:54,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-04-14 00:20:54,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VM OVFs in Data Center Lok >2014-04-14 00:20:54,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated VM OVFs in Data Center Lok >2014-04-14 00:20:54,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update template OVFs in Data Center Lok >2014-04-14 00:20:54,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated templates OVFs in Data Center Lok >2014-04-14 00:20:54,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 00:20:54,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 00:26:38,245 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 151418c2 >2014-04-14 00:26:38,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5a65735d >2014-04-14 00:26:38,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a65735d >2014-04-14 00:26:38,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 151418c2 >2014-04-14 00:26:38,486 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 01:20:52,784 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-04-14 01:20:54,127 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-04-14 01:20:54,128 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VM OVFs in Data Center Lok >2014-04-14 01:20:54,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated VM OVFs in Data Center Lok >2014-04-14 01:20:54,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update template OVFs in Data Center Lok >2014-04-14 01:20:54,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated templates OVFs in Data Center Lok >2014-04-14 01:20:54,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 01:20:54,130 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 01:26:38,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 101e7862 >2014-04-14 01:26:38,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 40717232 >2014-04-14 01:26:38,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 40717232 >2014-04-14 01:26:38,718 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 101e7862 >2014-04-14 01:26:38,723 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 02:20:52,786 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-04-14 02:20:54,131 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-14 02:20:54,132 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-14 02:20:54,132 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-14 02:20:54,133 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-14 02:20:54,133 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-14 02:20:54,133 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 02:20:54,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 02:26:38,728 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 538f708c >2014-04-14 02:26:38,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6e26222f >2014-04-14 02:26:38,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6e26222f >2014-04-14 02:26:38,949 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 538f708c >2014-04-14 02:26:38,955 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 03:20:52,787 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-04-14 03:20:54,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-14 03:20:54,135 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-04-14 03:20:54,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-04-14 03:20:54,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-04-14 03:20:54,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-04-14 03:20:54,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 03:20:54,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 03:26:38,959 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 536d4582 >2014-04-14 03:26:38,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2aee8498 >2014-04-14 03:26:39,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2aee8498 >2014-04-14 03:26:39,179 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 536d4582 >2014-04-14 03:26:39,184 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-37) Start deleteAgedOutAuditLogs >2014-04-14 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-37) Finished deleteAgedOutAuditLogs >2014-04-14 04:20:52,790 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-04-14 04:20:54,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-04-14 04:20:54,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VM OVFs in Data Center Lok >2014-04-14 04:20:54,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated VM OVFs in Data Center Lok >2014-04-14 04:20:54,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update template OVFs in Data Center Lok >2014-04-14 04:20:54,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated templates OVFs in Data Center Lok >2014-04-14 04:20:54,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 04:20:54,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 04:26:39,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6f1ac2d9 >2014-04-14 04:26:39,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 45ee6d8b >2014-04-14 04:26:39,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 45ee6d8b >2014-04-14 04:26:39,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6f1ac2d9 >2014-04-14 04:26:39,414 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 05:20:52,791 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-04-14 05:20:54,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-04-14 05:20:54,141 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Lok >2014-04-14 05:20:54,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Lok >2014-04-14 05:20:54,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Lok >2014-04-14 05:20:54,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated templates OVFs in Data Center Lok >2014-04-14 05:20:54,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 05:20:54,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 05:26:39,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7f05764c >2014-04-14 05:26:39,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3f99b8ba >2014-04-14 05:26:39,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3f99b8ba >2014-04-14 05:26:39,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f05764c >2014-04-14 05:26:39,643 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 06:20:52,792 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-5) Start refreshing all users data >2014-04-14 06:20:54,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VMs/Templates Ovf. >2014-04-14 06:20:54,144 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update VM OVFs in Data Center Lok >2014-04-14 06:20:54,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated VM OVFs in Data Center Lok >2014-04-14 06:20:54,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to update template OVFs in Data Center Lok >2014-04-14 06:20:54,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully updated templates OVFs in Data Center Lok >2014-04-14 06:20:54,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 06:20:54,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-89) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 06:26:39,646 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 59e8e2ae >2014-04-14 06:26:39,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 487a3b9f >2014-04-14 06:26:39,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 487a3b9f >2014-04-14 06:26:39,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 59e8e2ae >2014-04-14 06:26:39,871 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 07:20:52,794 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-04-14 07:20:54,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-04-14 07:20:54,150 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VM OVFs in Data Center Lok >2014-04-14 07:20:54,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated VM OVFs in Data Center Lok >2014-04-14 07:20:54,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update template OVFs in Data Center Lok >2014-04-14 07:20:54,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated templates OVFs in Data Center Lok >2014-04-14 07:20:54,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 07:20:54,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 07:26:39,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7e8b7937 >2014-04-14 07:26:39,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 76c34ac8 >2014-04-14 07:26:40,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76c34ac8 >2014-04-14 07:26:40,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7e8b7937 >2014-04-14 07:26:40,110 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 08:20:52,796 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-04-14 08:20:54,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-04-14 08:20:54,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VM OVFs in Data Center Lok >2014-04-14 08:20:54,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated VM OVFs in Data Center Lok >2014-04-14 08:20:54,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update template OVFs in Data Center Lok >2014-04-14 08:20:54,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated templates OVFs in Data Center Lok >2014-04-14 08:20:54,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 08:20:54,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 08:26:40,114 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3707831b >2014-04-14 08:26:40,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4e307dd6 >2014-04-14 08:26:40,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4e307dd6 >2014-04-14 08:26:40,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3707831b >2014-04-14 08:26:40,339 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 09:20:52,797 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-04-14 09:20:54,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-04-14 09:20:54,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VM OVFs in Data Center Lok >2014-04-14 09:20:54,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully updated VM OVFs in Data Center Lok >2014-04-14 09:20:54,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update template OVFs in Data Center Lok >2014-04-14 09:20:54,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully updated templates OVFs in Data Center Lok >2014-04-14 09:20:54,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 09:20:54,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 09:26:40,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4a406a4e >2014-04-14 09:26:40,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4b034cf7 >2014-04-14 09:26:40,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b034cf7 >2014-04-14 09:26:40,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4a406a4e >2014-04-14 09:26:40,590 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 10:20:52,799 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-14 10:20:54,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-04-14 10:20:54,159 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VM OVFs in Data Center Lok >2014-04-14 10:20:54,160 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated VM OVFs in Data Center Lok >2014-04-14 10:20:54,160 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update template OVFs in Data Center Lok >2014-04-14 10:20:54,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated templates OVFs in Data Center Lok >2014-04-14 10:20:54,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 10:20:54,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 10:26:40,594 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d5a7768 >2014-04-14 10:26:40,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1e555be6 >2014-04-14 10:26:40,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1e555be6 >2014-04-14 10:26:40,831 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d5a7768 >2014-04-14 10:26:40,837 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 11:20:52,801 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-04-14 11:20:54,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-04-14 11:20:54,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-04-14 11:20:54,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-04-14 11:20:54,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-04-14 11:20:54,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-04-14 11:20:54,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 11:20:54,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 11:26:40,841 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [4209317e] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 23a8dcd2 >2014-04-14 11:26:40,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [4209317e] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 23397148 >2014-04-14 11:26:41,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [4209317e] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23397148 >2014-04-14 11:26:41,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [4209317e] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23a8dcd2 >2014-04-14 11:26:41,189 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) [4209317e] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 12:20:52,803 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-04-14 12:20:54,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-04-14 12:20:54,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VM OVFs in Data Center Lok >2014-04-14 12:20:54,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated VM OVFs in Data Center Lok >2014-04-14 12:20:54,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update template OVFs in Data Center Lok >2014-04-14 12:20:54,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated templates OVFs in Data Center Lok >2014-04-14 12:20:54,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 12:20:54,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 12:26:41,192 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 27056cfc >2014-04-14 12:26:41,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 55362867 >2014-04-14 12:26:41,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 55362867 >2014-04-14 12:26:41,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 27056cfc >2014-04-14 12:26:41,595 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 13:20:52,805 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-04-14 13:20:54,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VMs/Templates Ovf. >2014-04-14 13:20:54,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update VM OVFs in Data Center Lok >2014-04-14 13:20:54,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Successfully updated VM OVFs in Data Center Lok >2014-04-14 13:20:54,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to update template OVFs in Data Center Lok >2014-04-14 13:20:54,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Successfully updated templates OVFs in Data Center Lok >2014-04-14 13:20:54,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 13:20:54,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 13:26:41,599 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7f5aeaf9 >2014-04-14 13:26:41,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 29f8c2c1 >2014-04-14 13:26:42,039 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 29f8c2c1 >2014-04-14 13:26:42,039 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f5aeaf9 >2014-04-14 13:26:42,045 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 13:36:01,139 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Loaded file "/usr/share/ovirt-engine/services/ovirt-engine/ovirt-engine.conf". >2014-04-14 13:36:01,154 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf". >2014-04-14 13:36:01,154 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/10-rhevm-doc.conf". >2014-04-14 13:36:01,154 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-database.conf". >2014-04-14 13:36:01,154 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-jboss.conf". >2014-04-14 13:36:01,154 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-pki.conf". >2014-04-14 13:36:01,154 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/10-setup-protocols.conf". >2014-04-14 13:36:01,155 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Loaded file "/etc/ovirt-engine/engine.conf.d/50-redhat-support-plugin.conf". >2014-04-14 13:36:01,157 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_AJP_ENABLED" is "true". >2014-04-14 13:36:01,157 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_AJP_PORT" is "8702". >2014-04-14 13:36:01,157 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_APPS" is "engine.ear redhat-support-plugin-rhev.ear". >2014-04-14 13:36:01,157 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_CACHE" is "/var/cache/ovirt-engine". >2014-04-14 13:36:01,157 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_CHECK_INTERVAL" is "1000". >2014-04-14 13:36:01,157 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_CONNECTION_TIMEOUT" is "300000". >2014-04-14 13:36:01,157 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_DATABASE" is "engine". >2014-04-14 13:36:01,157 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_DRIVER" is "org.postgresql.Driver". >2014-04-14 13:36:01,157 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_HOST" is "localhost". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_MAX_CONNECTIONS" is "100". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_MIN_CONNECTIONS" is "1". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_PASSWORD" is "***". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_PORT" is "5432". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_SECURED" is "False". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_SECURED_VALIDATION" is "False". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_SSL" is "false". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_URL" is "jdbc:postgresql://localhost:5432/engine?sslfactory=org.postgresql.ssl.NonValidatingFactory". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DB_USER" is "engine". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DEBUG_ADDRESS" is "". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_DOC" is "/usr/share/doc/ovirt-engine". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_ETC" is "/etc/ovirt-engine". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_FQDN" is "localhost.localdomain". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_GROUP" is "ovirt". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_HEAP_MAX" is "1g". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_HEAP_MIN" is "1g". >2014-04-14 13:36:01,158 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_HTTPS_ENABLED" is "false". >2014-04-14 13:36:01,159 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_HTTPS_PORT" is "None". >2014-04-14 13:36:01,159 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_HTTPS_PROTOCOLS" is "SSLv3,TLSv1,TLSv1.1,TLSv1.2". >2014-04-14 13:36:01,159 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_HTTP_ENABLED" is "false". >2014-04-14 13:36:01,159 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_HTTP_PORT" is "None". >2014-04-14 13:36:01,159 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_JVM_ARGS" is " -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="/var/log/ovirt-engine/dump"". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_LOG" is "/var/log/ovirt-engine". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_LOG_TO_CONSOLE" is "false". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_MANUAL" is "/usr/share/doc/rhevm-doc/manual". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PERM_MAX" is "256m". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PERM_MIN" is "256m". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI" is "/etc/pki/ovirt-engine". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_CA" is "/etc/pki/ovirt-engine/ca.pem". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_ENGINE_CERT" is "/etc/pki/ovirt-engine/certs/engine.cer". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_ENGINE_STORE" is "/etc/pki/ovirt-engine/keys/engine.p12". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_ENGINE_STORE_ALIAS" is "1". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_ENGINE_STORE_PASSWORD" is "***". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_TRUST_STORE" is "/etc/pki/ovirt-engine/.truststore". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PKI_TRUST_STORE_PASSWORD" is "***". >2014-04-14 13:36:01,174 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PROPERTIES" is " jsse.enableSNIExtension=false". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PROXY_ENABLED" is "true". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PROXY_HTTPS_PORT" is "443". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_PROXY_HTTP_PORT" is "80". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_STOP_INTERVAL" is "1". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_STOP_TIME" is "10". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_TMP" is "/var/tmp/ovirt-engine". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_UP_MARK" is "/var/lib/ovirt-engine/engine.up". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_USER" is "ovirt". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_USR" is "/usr/share/ovirt-engine". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_VAR" is "/var/lib/ovirt-engine". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "ENGINE_VERBOSE_GC" is "false". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "JBOSS_HOME" is "/usr/share/jbossas". >2014-04-14 13:36:01,175 INFO [org.ovirt.engine.core.utils.LocalConfig] (ServerService Thread Pool -- 46) Value of property "SENSITIVE_KEYS" is ",ENGINE_DB_PASSWORD,ENGINE_PKI_TRUST_STORE_PASSWORD,ENGINE_PKI_ENGINE_STORE_PASSWORD". >2014-04-14 13:36:02,405 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 47) Start initializing Backend >2014-04-14 13:36:02,891 ERROR [org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils] (ServerService Thread Pool -- 47) Failed to decrypt value for property LocalAdminPassword will be used encrypted value: javax.crypto.BadPaddingException: Data must start with zero > at sun.security.rsa.RSAPadding.unpadV15(RSAPadding.java:329) [rt.jar:1.7.0_51] > at sun.security.rsa.RSAPadding.unpad(RSAPadding.java:272) [rt.jar:1.7.0_51] > at com.sun.crypto.provider.RSACipher.doFinal(RSACipher.java:356) [sunjce_provider.jar:1.7.0_51] > at com.sun.crypto.provider.RSACipher.engineDoFinal(RSACipher.java:382) [sunjce_provider.jar:1.7.0_51] > at javax.crypto.Cipher.doFinal(Cipher.java:1922) [jce.jar:1.7.0_51] > at org.ovirt.engine.core.utils.crypt.EngineEncryptionUtils.decrypt(EngineEncryptionUtils.java:169) [utils.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.getValue(DBConfigUtils.java:107) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.updateOption(DBConfigUtils.java:275) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.refreshVdcOptionCache(DBConfigUtils.java:35) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.<init>(DBConfigUtils.java:166) [dal.jar:] > at org.ovirt.engine.core.dal.dbbroker.generic.DBConfigUtils.<init>(DBConfigUtils.java:152) [dal.jar:] > at org.ovirt.engine.core.bll.Backend.Initialize(Backend.java:183) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.create(Backend.java:130) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:130) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ManagedReferenceInterceptorFactory$ManagedReferenceInterceptor.processInvocation(ManagedReferenceInterceptorFactory.java:95) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.LifecycleCMTTxInterceptor.processInvocation(LifecycleCMTTxInterceptor.java:68) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:161) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:125) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.component.singleton.SingletonComponent.start(SingletonComponent.java:140) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ee.component.ComponentStartService$1.run(ComponentStartService.java:54) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [rt.jar:1.7.0_51] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > at org.jboss.threads.JBossThread.run(JBossThread.java:122) > >2014-04-14 13:36:03,014 WARN [org.ovirt.engine.core.utils.ConfigUtilsBase] (ServerService Thread Pool -- 47) Could not find enum value for option: WebAdminUpperRightButtonURL >2014-04-14 13:36:03,139 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 47) Running ovirt-engine 3.3.2-2.1.bz1083423.el6ev >2014-04-14 13:36:03,145 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 47) Start initializing dictionaries >2014-04-14 13:36:03,155 INFO [org.ovirt.engine.core.bll.CpuFlagsManagerHandler] (ServerService Thread Pool -- 47) Finished initializing dictionaries >2014-04-14 13:36:03,156 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 47) Start initializing AuditLogCleanupManager >2014-04-14 13:36:03,156 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 47) Setting audit cleanup manager to run at: 35 35 3 * * ? >2014-04-14 13:36:03,190 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (ServerService Thread Pool -- 47) Finished initializing AuditLogCleanupManager >2014-04-14 13:36:03,195 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 47) Start initializing TagsDirector >2014-04-14 13:36:03,216 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 47) Tag root added to tree >2014-04-14 13:36:03,239 INFO [org.ovirt.engine.core.bll.TagsDirector] (ServerService Thread Pool -- 47) Finished initializing TagsDirector >2014-04-14 13:36:03,241 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 47) Start initializing IsoDomainListSyncronizer >2014-04-14 13:36:03,251 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (ServerService Thread Pool -- 47) Finished initializing IsoDomainListSyncronizer >2014-04-14 13:36:03,767 INFO [org.ovirt.engine.core.utils.osinfo.OsInfoPreferencesLoader] (ServerService Thread Pool -- 47) Loaded file /etc/ovirt-engine/osinfo.conf.d/00-defaults.properties >2014-04-14 13:36:04,040 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 47) Completed initializing handlers >2014-04-14 13:36:04,041 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 47) Start initializing ErrorTranslatorImpl >2014-04-14 13:36:04,068 WARN [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 47) Code MAC_ADDRESS_IS_IN_USE appears more than once in string table. >2014-04-14 13:36:04,069 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 47) Finished initializing ErrorTranslatorImpl >2014-04-14 13:36:04,069 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 47) Start initializing ErrorTranslatorImpl >2014-04-14 13:36:04,070 INFO [org.ovirt.engine.core.utils.ErrorTranslatorImpl] (ServerService Thread Pool -- 47) Finished initializing ErrorTranslatorImpl >2014-04-14 13:36:04,070 INFO [org.ovirt.engine.core.bll.Backend] (ServerService Thread Pool -- 47) Mark incomplete jobs as UNKNOWN >2014-04-14 13:36:04,130 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 47) Start initializing JobRepositoryCleanupManager >2014-04-14 13:36:04,132 INFO [org.ovirt.engine.core.bll.job.JobRepositoryCleanupManager] (ServerService Thread Pool -- 47) Finished initializing JobRepositoryCleanupManager >2014-04-14 13:36:04,133 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 47) Start initializing AutoRecoveryManager >2014-04-14 13:36:04,135 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (ServerService Thread Pool -- 47) Finished initializing AutoRecoveryManager >2014-04-14 13:36:04,137 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 47) Start initializing ExecutionMessageDirector >2014-04-14 13:36:04,138 INFO [org.ovirt.engine.core.dal.job.ExecutionMessageDirector] (ServerService Thread Pool -- 47) Finished initializing ExecutionMessageDirector >2014-04-14 13:36:04,172 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 36) Start initializing UsersDomainsCacheManagerService >2014-04-14 13:36:04,201 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (ServerService Thread Pool -- 36) Start initializing DbUserCacheManager >2014-04-14 13:36:04,211 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (ServerService Thread Pool -- 36) Finished initializing DbUserCacheManager >2014-04-14 13:36:04,211 INFO [org.ovirt.engine.core.bll.adbroker.UsersDomainsCacheManagerService] (ServerService Thread Pool -- 36) Finished initializing UsersDomainsCacheManagerService >2014-04-14 13:36:04,217 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (ServerService Thread Pool -- 47) Initialization of AsyncTaskManager completed successfully. >2014-04-14 13:36:04,224 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) Start initializing ResourceManager >2014-04-14 13:36:04,264 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-04-14 13:36:04,514 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 47) Entered VdsManager constructor >2014-04-14 13:36:04,587 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ServerService Thread Pool -- 47) Initialize vdsBroker (10.34.27.209,54,321) >2014-04-14 13:36:04,913 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) VDS 53fd133a-9dc6-11e2-abd9-001a4a223991 was added to the Resource Manager >2014-04-14 13:36:04,956 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ServerService Thread Pool -- 47) Finished initializing ResourceManager >2014-04-14 13:36:04,970 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (ServerService Thread Pool -- 47) Initialization of OvfDataUpdater completed successfully. >2014-04-14 13:36:04,978 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Start scheduling to enable vds load balancer >2014-04-14 13:36:04,979 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Finished scheduling to enable vds load balancer >2014-04-14 13:36:04,996 INFO [org.ovirt.engine.core.bll.network.MacPoolManager] (pool-4-thread-1) MacPoolManager(17d36f6): Start initializing >2014-04-14 13:36:05,034 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 47) Init VM custom properties utilities >2014-04-14 13:36:05,039 INFO [org.ovirt.engine.core.bll.InitBackendServicesOnStartupBean] (ServerService Thread Pool -- 47) Init device custom properties utilities >2014-04-14 13:36:05,047 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Initializing Scheduling manager >2014-04-14 13:36:05,078 INFO [org.ovirt.engine.core.bll.network.MacPoolManager] (pool-4-thread-1) MacPoolManager(17d36f6): Finished initializing. Available MACs in pool: 253 >2014-04-14 13:36:05,129 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) External scheduler disabled, discovery skipped >2014-04-14 13:36:05,129 INFO [org.ovirt.engine.core.bll.scheduling.SchedulingManager] (ServerService Thread Pool -- 47) Initialized Scheduling manager >2014-04-14 13:36:05,130 INFO [org.ovirt.engine.core.bll.dwh.DwhHeartBeat] (ServerService Thread Pool -- 47) Initializing DWH Heart Beat >2014-04-14 13:36:10,211 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3a3ae5f9 >2014-04-14 13:36:10,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) FINISH, GetHardwareInfoVDSCommand, log id: 3a3ae5f9 >2014-04-14 13:36:10,635 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: UNASSIGNED not exist in string table >2014-04-14 13:36:10,635 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_FAILED_REMOVE_VM not exist in string table >2014-04-14 13:36:10,635 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not exist in string table >2014-04-14 13:36:10,636 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not exist in string table >2014-04-14 13:36:10,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: STORAGE_ACTIVATE_ASYNC not exist in string table >2014-04-14 13:36:10,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: DWH_STOPPED not exist in string table >2014-04-14 13:36:10,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: DWH_STARTED not exist in string table >2014-04-14 13:36:10,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: DWH_ERROR not exist in string table >2014-04-14 13:36:10,658 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_FAILED_REMOVE_VM not have severity. Assumed Normal >2014-04-14 13:36:10,658 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_ATTACH_DISK_TO_VM not have severity. Assumed Normal >2014-04-14 13:36:10,658 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2014-04-14 13:36:10,659 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_FAILED_DETACH_DISK_FROM_VM not have severity. Assumed Normal >2014-04-14 13:36:10,659 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_RUN_UNLOCK_ENTITY_SCRIPT not have severity. Assumed Normal >2014-04-14 13:36:10,659 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: GLUSTER_VOLUME_OPTION_CHANGED_FROM_CLI not have severity. Assumed Normal >2014-04-14 13:36:10,659 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: GLUSTER_SERVICES_LIST_NOT_FETCHED not have severity. Assumed Normal >2014-04-14 13:36:10,659 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: GLUSTER_VOLUME_BRICK_ADDED not have severity. Assumed Normal >2014-04-14 13:36:10,659 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_EXTEND_DISK_SIZE_UPDATE_VM_FAILURE not have severity. Assumed Normal >2014-04-14 13:36:10,663 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: VM_MIGRATION_START_SYSTEM_INITIATED not have severity. Assumed Normal >2014-04-14 13:36:10,663 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: VDS_NETWORK_MTU_DIFFER_FROM_LOGICAL_NETWORK not have severity. Assumed Normal >2014-04-14 13:36:10,663 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_UP not have severity. Assumed Normal >2014-04-14 13:36:10,663 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: NETWORK_UPDATE_VM_INTERFACE_LINK_DOWN not have severity. Assumed Normal >2014-04-14 13:36:10,663 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_UPDATED_MOM_POLICIES not have severity. Assumed Normal >2014-04-14 13:36:10,663 WARN [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) AuditLogType: USER_FAILED_TO_UPDATE_MOM_POLICIES not have severity. Assumed Normal >2014-04-14 13:36:10,733 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-14 13:36:11,274 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-3) Initializing Host: 10.34.27.209 >2014-04-14 13:36:11,497 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-14 13:36:14,980 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-14 13:36:15,305 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Correlation ID: 6d63ed15, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Lok. Setting status to Non Responsive. >2014-04-14 13:36:15,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-8) [6d63ed15] hostFromVds::selectedVds - 10.34.27.209, spmStatus SPM, storage pool Lok >2014-04-14 13:36:15,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Initialize Irs proxy from vds: 10.34.27.209 >2014-04-14 13:36:15,821 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host 10.34.27.209 (Address: 10.34.27.209). >2014-04-14 13:36:15,899 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-10) [6d63ed15] START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 19a4f267 >2014-04-14 13:36:16,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-10) [6d63ed15] -- executeIrsBrokerCommand: Attempting on storage pool 19dcb0b3-4c52-4803-a39e-992b1392da62 >2014-04-14 13:36:16,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (pool-4-thread-10) [6d63ed15] START, HSMGetAllTasksInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3de1a9c5 >2014-04-14 13:36:16,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (pool-4-thread-10) [6d63ed15] FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 3de1a9c5 >2014-04-14 13:36:16,829 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-10) [6d63ed15] FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 19a4f267 >2014-04-14 13:36:16,829 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-10) [6d63ed15] Discovered no tasks on Storage Pool Lok >2014-04-14 13:39:01,764 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-5) Reading UI plugin descriptor [/usr/share/ovirt-engine/ui-plugins/redhat_support_plugin_rhev.json] >2014-04-14 13:39:01,777 INFO [org.ovirt.engine.ui.frontend.server.gwt.plugin.PluginDataManager] (ajp-/127.0.0.1:8702-5) Reading UI plugin configuration [/etc/ovirt-engine/ui-plugins/redhat_support_plugin_rhev-config.json] >2014-04-14 13:39:08,826 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginAdminUserCommand internal: false. >2014-04-14 13:39:08,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-14 13:39:10,592 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-2) Running command: LoginUserCommand internal: false. >2014-04-14 13:39:10,607 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-14 14:36:04,295 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-04-14 14:36:04,969 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-14 14:36:04,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-14 14:36:04,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-14 14:36:04,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-14 14:36:04,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-14 14:36:04,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 14:36:05,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 14:41:03,272 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 153a626 >2014-04-14 14:41:03,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3455c557 >2014-04-14 14:41:03,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3455c557 >2014-04-14 14:41:03,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 153a626 >2014-04-14 14:41:03,690 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 15:36:04,299 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-04-14 15:36:05,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-04-14 15:36:05,005 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-04-14 15:36:05,015 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-04-14 15:36:05,015 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-04-14 15:36:05,020 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-04-14 15:36:05,020 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 15:36:05,021 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 15:41:03,699 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5fd48bb2 >2014-04-14 15:41:03,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7b1f8444 >2014-04-14 15:41:03,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7b1f8444 >2014-04-14 15:41:03,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5fd48bb2 >2014-04-14 15:41:03,974 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 16:36:04,302 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-04-14 16:36:05,022 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-14 16:36:05,024 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-04-14 16:36:05,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-04-14 16:36:05,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-04-14 16:36:05,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-04-14 16:36:05,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 16:36:05,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 16:41:03,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4bc4f381 >2014-04-14 16:41:04,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7ca76e1f >2014-04-14 16:41:04,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7ca76e1f >2014-04-14 16:41:04,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4bc4f381 >2014-04-14 16:41:04,559 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 17:36:04,307 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-04-14 17:36:05,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-04-14 17:36:05,031 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VM OVFs in Data Center Lok >2014-04-14 17:36:05,032 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated VM OVFs in Data Center Lok >2014-04-14 17:36:05,032 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update template OVFs in Data Center Lok >2014-04-14 17:36:05,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated templates OVFs in Data Center Lok >2014-04-14 17:36:05,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 17:36:05,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 17:41:04,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 60dd87b9 >2014-04-14 17:41:04,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6ff9b45d >2014-04-14 17:41:04,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ff9b45d >2014-04-14 17:41:04,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 60dd87b9 >2014-04-14 17:41:04,851 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 18:36:04,312 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-04-14 18:36:05,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-04-14 18:36:05,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VM OVFs in Data Center Lok >2014-04-14 18:36:05,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated VM OVFs in Data Center Lok >2014-04-14 18:36:05,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update template OVFs in Data Center Lok >2014-04-14 18:36:05,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated templates OVFs in Data Center Lok >2014-04-14 18:36:05,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 18:36:05,049 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 18:41:04,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5e61a0d1 >2014-04-14 18:41:04,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5b8529bf >2014-04-14 18:41:05,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5b8529bf >2014-04-14 18:41:05,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e61a0d1 >2014-04-14 18:41:05,348 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 19:36:04,313 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-04-14 19:36:05,049 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-04-14 19:36:05,051 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VM OVFs in Data Center Lok >2014-04-14 19:36:05,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated VM OVFs in Data Center Lok >2014-04-14 19:36:05,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update template OVFs in Data Center Lok >2014-04-14 19:36:05,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated templates OVFs in Data Center Lok >2014-04-14 19:36:05,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 19:36:05,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 19:41:05,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4e40b663 >2014-04-14 19:41:05,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7a8645b7 >2014-04-14 19:41:05,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7a8645b7 >2014-04-14 19:41:05,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4e40b663 >2014-04-14 19:41:05,597 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 20:36:04,315 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-04-14 20:36:05,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-04-14 20:36:05,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VM OVFs in Data Center Lok >2014-04-14 20:36:05,060 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated VM OVFs in Data Center Lok >2014-04-14 20:36:05,060 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update template OVFs in Data Center Lok >2014-04-14 20:36:05,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated templates OVFs in Data Center Lok >2014-04-14 20:36:05,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 20:36:05,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 20:41:05,602 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 31e4403f >2014-04-14 20:41:05,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3601b0 >2014-04-14 20:41:06,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3601b0 >2014-04-14 20:41:06,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 31e4403f >2014-04-14 20:41:06,138 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 21:36:04,316 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-04-14 21:36:05,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-04-14 21:36:05,064 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Lok >2014-04-14 21:36:05,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Lok >2014-04-14 21:36:05,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Lok >2014-04-14 21:36:05,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated templates OVFs in Data Center Lok >2014-04-14 21:36:05,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 21:36:05,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 21:41:06,143 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 717e1ce6 >2014-04-14 21:41:06,145 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: e9b7a40 >2014-04-14 21:41:06,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: e9b7a40 >2014-04-14 21:41:06,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 717e1ce6 >2014-04-14 21:41:06,371 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 22:36:04,318 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-14 22:36:05,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update VMs/Templates Ovf. >2014-04-14 22:36:05,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update VM OVFs in Data Center Lok >2014-04-14 22:36:05,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully updated VM OVFs in Data Center Lok >2014-04-14 22:36:05,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update template OVFs in Data Center Lok >2014-04-14 22:36:05,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully updated templates OVFs in Data Center Lok >2014-04-14 22:36:05,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 22:36:05,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 22:41:06,377 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3efdab77 >2014-04-14 22:41:06,378 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 431f0 >2014-04-14 22:41:06,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 431f0 >2014-04-14 22:41:06,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3efdab77 >2014-04-14 22:41:06,611 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-14 23:36:04,323 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-14 23:36:05,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-04-14 23:36:05,079 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VM OVFs in Data Center Lok >2014-04-14 23:36:05,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated VM OVFs in Data Center Lok >2014-04-14 23:36:05,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update template OVFs in Data Center Lok >2014-04-14 23:36:05,081 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated templates OVFs in Data Center Lok >2014-04-14 23:36:05,081 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-14 23:36:05,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-14 23:41:06,615 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 15d10a1e >2014-04-14 23:41:06,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4eef42c2 >2014-04-14 23:41:07,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4eef42c2 >2014-04-14 23:41:07,112 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 15d10a1e >2014-04-14 23:41:07,118 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 00:36:04,325 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-04-15 00:36:05,083 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-04-15 00:36:05,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VM OVFs in Data Center Lok >2014-04-15 00:36:05,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated VM OVFs in Data Center Lok >2014-04-15 00:36:05,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update template OVFs in Data Center Lok >2014-04-15 00:36:05,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated templates OVFs in Data Center Lok >2014-04-15 00:36:05,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 00:36:05,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 00:41:07,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3acdae04 >2014-04-15 00:41:07,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 155ba6c5 >2014-04-15 00:41:07,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 155ba6c5 >2014-04-15 00:41:07,709 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3acdae04 >2014-04-15 00:41:07,715 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 01:36:04,327 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-04-15 01:36:05,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-04-15 01:36:05,087 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VM OVFs in Data Center Lok >2014-04-15 01:36:05,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Successfully updated VM OVFs in Data Center Lok >2014-04-15 01:36:05,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update template OVFs in Data Center Lok >2014-04-15 01:36:05,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Successfully updated templates OVFs in Data Center Lok >2014-04-15 01:36:05,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 01:36:05,089 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 01:41:07,719 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [6d63ed15] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3b0fc3d0 >2014-04-15 01:41:07,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [6d63ed15] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: d27867e >2014-04-15 01:41:08,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [6d63ed15] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: d27867e >2014-04-15 01:41:08,159 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [6d63ed15] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3b0fc3d0 >2014-04-15 01:41:08,169 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) [6d63ed15] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 02:36:04,330 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-04-15 02:36:05,089 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-04-15 02:36:05,090 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VM OVFs in Data Center Lok >2014-04-15 02:36:05,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated VM OVFs in Data Center Lok >2014-04-15 02:36:05,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update template OVFs in Data Center Lok >2014-04-15 02:36:05,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated templates OVFs in Data Center Lok >2014-04-15 02:36:05,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 02:36:05,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 02:41:08,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7455e859 >2014-04-15 02:41:08,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 340966ba >2014-04-15 02:41:08,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 340966ba >2014-04-15 02:41:08,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7455e859 >2014-04-15 02:41:08,416 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) Start deleteAgedOutAuditLogs >2014-04-15 03:35:35,009 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-46) Finished deleteAgedOutAuditLogs >2014-04-15 03:36:04,332 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-15 03:36:05,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-04-15 03:36:05,093 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VM OVFs in Data Center Lok >2014-04-15 03:36:05,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated VM OVFs in Data Center Lok >2014-04-15 03:36:05,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update template OVFs in Data Center Lok >2014-04-15 03:36:05,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated templates OVFs in Data Center Lok >2014-04-15 03:36:05,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 03:36:05,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 03:41:08,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2974d6cd >2014-04-15 03:41:08,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 35c241fe >2014-04-15 03:41:08,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 35c241fe >2014-04-15 03:41:08,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2974d6cd >2014-04-15 03:41:08,654 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 04:36:04,336 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-04-15 04:36:05,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-15 04:36:05,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-15 04:36:05,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-15 04:36:05,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-15 04:36:05,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-15 04:36:05,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 04:36:05,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 04:41:08,659 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 698248e8 >2014-04-15 04:41:08,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6880ad75 >2014-04-15 04:41:08,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6880ad75 >2014-04-15 04:41:08,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 698248e8 >2014-04-15 04:41:08,881 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 05:36:04,338 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-81) Start refreshing all users data >2014-04-15 05:36:05,100 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-04-15 05:36:05,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VM OVFs in Data Center Lok >2014-04-15 05:36:05,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated VM OVFs in Data Center Lok >2014-04-15 05:36:05,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update template OVFs in Data Center Lok >2014-04-15 05:36:05,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated templates OVFs in Data Center Lok >2014-04-15 05:36:05,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 05:36:05,103 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 05:41:08,883 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 78483f06 >2014-04-15 05:41:08,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4f21e09b >2014-04-15 05:41:09,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f21e09b >2014-04-15 05:41:09,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 78483f06 >2014-04-15 05:41:09,111 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 06:36:04,340 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-04-15 06:36:05,103 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-04-15 06:36:05,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VM OVFs in Data Center Lok >2014-04-15 06:36:05,105 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated VM OVFs in Data Center Lok >2014-04-15 06:36:05,105 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update template OVFs in Data Center Lok >2014-04-15 06:36:05,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated templates OVFs in Data Center Lok >2014-04-15 06:36:05,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 06:36:05,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 06:41:09,115 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7ae3a58b >2014-04-15 06:41:09,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 778987b4 >2014-04-15 06:41:09,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 778987b4 >2014-04-15 06:41:09,333 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7ae3a58b >2014-04-15 06:41:09,339 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 07:36:04,342 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-04-15 07:36:05,106 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-04-15 07:36:05,107 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VM OVFs in Data Center Lok >2014-04-15 07:36:05,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated VM OVFs in Data Center Lok >2014-04-15 07:36:05,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update template OVFs in Data Center Lok >2014-04-15 07:36:05,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated templates OVFs in Data Center Lok >2014-04-15 07:36:05,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 07:36:05,109 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 07:41:09,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 971073e >2014-04-15 07:41:09,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5b927c49 >2014-04-15 07:41:09,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5b927c49 >2014-04-15 07:41:09,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 971073e >2014-04-15 07:41:09,570 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 08:36:04,344 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-04-15 08:36:05,109 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-15 08:36:05,110 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-15 08:36:05,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-15 08:36:05,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-15 08:36:05,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-15 08:36:05,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 08:36:05,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 08:41:09,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 282a49e5 >2014-04-15 08:41:09,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 70596dfc >2014-04-15 08:41:09,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 70596dfc >2014-04-15 08:41:09,822 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 282a49e5 >2014-04-15 08:41:09,829 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 09:36:04,346 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-04-15 09:36:05,113 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-04-15 09:36:05,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VM OVFs in Data Center Lok >2014-04-15 09:36:05,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated VM OVFs in Data Center Lok >2014-04-15 09:36:05,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update template OVFs in Data Center Lok >2014-04-15 09:36:05,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated templates OVFs in Data Center Lok >2014-04-15 09:36:05,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 09:36:05,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 09:41:09,833 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 66ea94c >2014-04-15 09:41:09,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4b2cbfe0 >2014-04-15 09:41:10,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b2cbfe0 >2014-04-15 09:41:10,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 66ea94c >2014-04-15 09:41:10,063 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 10:36:04,348 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-04-15 10:36:05,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-04-15 10:36:05,122 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VM OVFs in Data Center Lok >2014-04-15 10:36:05,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated VM OVFs in Data Center Lok >2014-04-15 10:36:05,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update template OVFs in Data Center Lok >2014-04-15 10:36:05,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated templates OVFs in Data Center Lok >2014-04-15 10:36:05,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 10:36:05,124 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 10:41:10,067 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 46145bd2 >2014-04-15 10:41:10,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 62985bef >2014-04-15 10:41:10,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 62985bef >2014-04-15 10:41:10,292 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 46145bd2 >2014-04-15 10:41:10,296 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 11:36:04,350 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-04-15 11:36:05,124 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-04-15 11:36:05,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VM OVFs in Data Center Lok >2014-04-15 11:36:05,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated VM OVFs in Data Center Lok >2014-04-15 11:36:05,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update template OVFs in Data Center Lok >2014-04-15 11:36:05,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated templates OVFs in Data Center Lok >2014-04-15 11:36:05,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 11:36:05,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 11:41:10,300 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2c7c64ff >2014-04-15 11:41:10,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1acab82a >2014-04-15 11:41:10,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1acab82a >2014-04-15 11:41:10,546 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2c7c64ff >2014-04-15 11:41:10,570 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 12:36:04,351 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-04-15 12:36:05,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-04-15 12:36:05,127 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VM OVFs in Data Center Lok >2014-04-15 12:36:05,127 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated VM OVFs in Data Center Lok >2014-04-15 12:36:05,127 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update template OVFs in Data Center Lok >2014-04-15 12:36:05,128 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated templates OVFs in Data Center Lok >2014-04-15 12:36:05,128 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 12:36:05,128 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 12:41:10,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 38687395 >2014-04-15 12:41:10,574 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 517c5821 >2014-04-15 12:41:10,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 517c5821 >2014-04-15 12:41:10,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 38687395 >2014-04-15 12:41:10,806 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 13:36:04,353 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-04-15 13:36:05,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-04-15 13:36:05,130 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VM OVFs in Data Center Lok >2014-04-15 13:36:05,130 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated VM OVFs in Data Center Lok >2014-04-15 13:36:05,130 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update template OVFs in Data Center Lok >2014-04-15 13:36:05,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated templates OVFs in Data Center Lok >2014-04-15 13:36:05,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 13:36:05,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 13:41:10,810 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2473ed46 >2014-04-15 13:41:10,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 36dea15c >2014-04-15 13:41:11,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 36dea15c >2014-04-15 13:41:11,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2473ed46 >2014-04-15 13:41:11,326 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 14:36:04,354 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-04-15 14:36:05,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-04-15 14:36:05,135 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-04-15 14:36:05,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-04-15 14:36:05,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-04-15 14:36:05,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-04-15 14:36:05,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 14:36:05,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 14:41:11,331 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7f4fcfd9 >2014-04-15 14:41:11,332 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 50d6f0db >2014-04-15 14:41:11,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 50d6f0db >2014-04-15 14:41:11,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f4fcfd9 >2014-04-15 14:41:11,872 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 15:36:04,356 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-04-15 15:36:05,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-04-15 15:36:05,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VM OVFs in Data Center Lok >2014-04-15 15:36:05,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully updated VM OVFs in Data Center Lok >2014-04-15 15:36:05,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update template OVFs in Data Center Lok >2014-04-15 15:36:05,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully updated templates OVFs in Data Center Lok >2014-04-15 15:36:05,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 15:36:05,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 15:41:11,875 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 15eecf95 >2014-04-15 15:41:11,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 717e4a72 >2014-04-15 15:41:12,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 717e4a72 >2014-04-15 15:41:12,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 15eecf95 >2014-04-15 15:41:12,106 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 16:36:04,357 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-04-15 16:36:05,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-04-15 16:36:05,141 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VM OVFs in Data Center Lok >2014-04-15 16:36:05,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated VM OVFs in Data Center Lok >2014-04-15 16:36:05,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update template OVFs in Data Center Lok >2014-04-15 16:36:05,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated templates OVFs in Data Center Lok >2014-04-15 16:36:05,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 16:36:05,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 16:41:12,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7a5c44be >2014-04-15 16:41:12,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3fa554b4 >2014-04-15 16:41:12,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3fa554b4 >2014-04-15 16:41:12,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7a5c44be >2014-04-15 16:41:12,338 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 17:36:04,358 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-89) Start refreshing all users data >2014-04-15 17:36:05,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-04-15 17:36:05,144 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VM OVFs in Data Center Lok >2014-04-15 17:36:05,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully updated VM OVFs in Data Center Lok >2014-04-15 17:36:05,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update template OVFs in Data Center Lok >2014-04-15 17:36:05,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully updated templates OVFs in Data Center Lok >2014-04-15 17:36:05,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 17:36:05,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 17:41:12,341 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 335dc13 >2014-04-15 17:41:12,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4503650a >2014-04-15 17:41:17,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4503650a >2014-04-15 17:41:17,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 335dc13 >2014-04-15 17:41:17,870 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 18:36:04,359 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-15 18:36:05,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VMs/Templates Ovf. >2014-04-15 18:36:05,147 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update VM OVFs in Data Center Lok >2014-04-15 18:36:05,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated VM OVFs in Data Center Lok >2014-04-15 18:36:05,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to update template OVFs in Data Center Lok >2014-04-15 18:36:05,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully updated templates OVFs in Data Center Lok >2014-04-15 18:36:05,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 18:36:05,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-15) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 18:41:17,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 402e764d >2014-04-15 18:41:17,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2a58da26 >2014-04-15 18:41:18,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2a58da26 >2014-04-15 18:41:18,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 402e764d >2014-04-15 18:41:18,124 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 19:36:04,361 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-04-15 19:36:05,150 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-04-15 19:36:05,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VM OVFs in Data Center Lok >2014-04-15 19:36:05,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully updated VM OVFs in Data Center Lok >2014-04-15 19:36:05,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update template OVFs in Data Center Lok >2014-04-15 19:36:05,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully updated templates OVFs in Data Center Lok >2014-04-15 19:36:05,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 19:36:05,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 19:41:18,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4a36ac6e >2014-04-15 19:41:18,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4b728fd2 >2014-04-15 19:41:18,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b728fd2 >2014-04-15 19:41:18,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4a36ac6e >2014-04-15 19:41:18,354 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 20:36:04,363 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-15 20:36:05,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-15 20:36:05,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-04-15 20:36:05,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-04-15 20:36:05,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-04-15 20:36:05,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-04-15 20:36:05,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 20:36:05,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 20:41:18,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2ad3676f >2014-04-15 20:41:18,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1d2bbe60 >2014-04-15 20:41:18,852 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d2bbe60 >2014-04-15 20:41:18,852 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2ad3676f >2014-04-15 20:41:18,856 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 21:36:04,364 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-04-15 21:36:05,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-04-15 21:36:05,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-04-15 21:36:05,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-04-15 21:36:05,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-04-15 21:36:05,159 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-04-15 21:36:05,159 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 21:36:05,159 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 21:41:18,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3dcfb287 >2014-04-15 21:41:18,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: ed1dcc5 >2014-04-15 21:41:19,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ed1dcc5 >2014-04-15 21:41:19,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3dcfb287 >2014-04-15 21:41:19,095 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 22:36:04,366 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-04-15 22:36:05,159 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-04-15 22:36:05,160 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VM OVFs in Data Center Lok >2014-04-15 22:36:05,161 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated VM OVFs in Data Center Lok >2014-04-15 22:36:05,161 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update template OVFs in Data Center Lok >2014-04-15 22:36:05,162 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated templates OVFs in Data Center Lok >2014-04-15 22:36:05,162 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 22:36:05,163 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 22:41:19,101 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1dbc05dd >2014-04-15 22:41:19,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2324ff6b >2014-04-15 22:41:19,337 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2324ff6b >2014-04-15 22:41:19,338 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1dbc05dd >2014-04-15 22:41:19,342 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-15 23:36:04,367 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-04-15 23:36:05,164 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-04-15 23:36:05,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VM OVFs in Data Center Lok >2014-04-15 23:36:05,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated VM OVFs in Data Center Lok >2014-04-15 23:36:05,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update template OVFs in Data Center Lok >2014-04-15 23:36:05,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated templates OVFs in Data Center Lok >2014-04-15 23:36:05,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-15 23:36:05,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-15 23:41:19,346 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1dbdb145 >2014-04-15 23:41:19,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 10c13844 >2014-04-15 23:41:19,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 10c13844 >2014-04-15 23:41:19,571 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1dbdb145 >2014-04-15 23:41:19,575 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 00:36:04,369 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-04-16 00:36:05,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-04-16 00:36:05,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VM OVFs in Data Center Lok >2014-04-16 00:36:05,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated VM OVFs in Data Center Lok >2014-04-16 00:36:05,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update template OVFs in Data Center Lok >2014-04-16 00:36:05,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated templates OVFs in Data Center Lok >2014-04-16 00:36:05,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 00:36:05,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 00:41:19,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6584fdc0 >2014-04-16 00:41:19,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7e2f7c51 >2014-04-16 00:41:19,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7e2f7c51 >2014-04-16 00:41:19,800 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6584fdc0 >2014-04-16 00:41:19,819 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 01:36:04,371 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-04-16 01:36:05,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-04-16 01:36:05,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VM OVFs in Data Center Lok >2014-04-16 01:36:05,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated VM OVFs in Data Center Lok >2014-04-16 01:36:05,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update template OVFs in Data Center Lok >2014-04-16 01:36:05,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated templates OVFs in Data Center Lok >2014-04-16 01:36:05,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 01:36:05,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 01:41:19,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 46a3f466 >2014-04-16 01:41:19,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 360278c5 >2014-04-16 01:41:20,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 360278c5 >2014-04-16 01:41:20,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 46a3f466 >2014-04-16 01:41:20,215 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 02:36:04,373 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-04-16 02:36:05,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-04-16 02:36:05,175 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VM OVFs in Data Center Lok >2014-04-16 02:36:05,176 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully updated VM OVFs in Data Center Lok >2014-04-16 02:36:05,176 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update template OVFs in Data Center Lok >2014-04-16 02:36:05,176 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully updated templates OVFs in Data Center Lok >2014-04-16 02:36:05,176 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 02:36:05,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 02:41:20,218 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 22ca0893 >2014-04-16 02:41:20,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7df8fcbc >2014-04-16 02:41:20,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7df8fcbc >2014-04-16 02:41:20,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 22ca0893 >2014-04-16 02:41:20,445 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-28) Start deleteAgedOutAuditLogs >2014-04-16 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-28) Finished deleteAgedOutAuditLogs >2014-04-16 03:36:04,373 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-04-16 03:36:05,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-04-16 03:36:05,178 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VM OVFs in Data Center Lok >2014-04-16 03:36:05,179 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated VM OVFs in Data Center Lok >2014-04-16 03:36:05,179 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update template OVFs in Data Center Lok >2014-04-16 03:36:05,179 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated templates OVFs in Data Center Lok >2014-04-16 03:36:05,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 03:36:05,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 03:41:20,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 33ba2b6f >2014-04-16 03:41:20,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3799ef1 >2014-04-16 03:41:20,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3799ef1 >2014-04-16 03:41:20,674 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 33ba2b6f >2014-04-16 03:41:20,679 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 04:36:04,376 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-04-16 04:36:05,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-04-16 04:36:05,181 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VM OVFs in Data Center Lok >2014-04-16 04:36:05,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated VM OVFs in Data Center Lok >2014-04-16 04:36:05,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update template OVFs in Data Center Lok >2014-04-16 04:36:05,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated templates OVFs in Data Center Lok >2014-04-16 04:36:05,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 04:36:05,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 04:41:20,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3e4de035 >2014-04-16 04:41:20,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6abad787 >2014-04-16 04:41:20,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6abad787 >2014-04-16 04:41:20,912 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3e4de035 >2014-04-16 04:41:20,919 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 05:36:04,378 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-04-16 05:36:05,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-04-16 05:36:05,187 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VM OVFs in Data Center Lok >2014-04-16 05:36:05,191 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated VM OVFs in Data Center Lok >2014-04-16 05:36:05,191 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update template OVFs in Data Center Lok >2014-04-16 05:36:05,191 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated templates OVFs in Data Center Lok >2014-04-16 05:36:05,191 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 05:36:05,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 05:41:20,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4331896f >2014-04-16 05:41:20,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6935f8df >2014-04-16 05:41:21,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6935f8df >2014-04-16 05:41:21,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4331896f >2014-04-16 05:41:21,154 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 06:36:04,380 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-04-16 06:36:05,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-04-16 06:36:05,193 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VM OVFs in Data Center Lok >2014-04-16 06:36:05,194 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated VM OVFs in Data Center Lok >2014-04-16 06:36:05,194 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update template OVFs in Data Center Lok >2014-04-16 06:36:05,195 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated templates OVFs in Data Center Lok >2014-04-16 06:36:05,195 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 06:36:05,195 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 06:41:21,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4cf9904 >2014-04-16 06:41:21,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5946d813 >2014-04-16 06:41:21,380 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5946d813 >2014-04-16 06:41:21,380 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4cf9904 >2014-04-16 06:41:21,384 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 07:36:04,382 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-16 07:36:05,195 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-04-16 07:36:05,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Lok >2014-04-16 07:36:05,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Lok >2014-04-16 07:36:05,196 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Lok >2014-04-16 07:36:05,197 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated templates OVFs in Data Center Lok >2014-04-16 07:36:05,197 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 07:36:05,197 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 07:41:21,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 16402c94 >2014-04-16 07:41:21,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4f2eb3f5 >2014-04-16 07:41:21,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f2eb3f5 >2014-04-16 07:41:21,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 16402c94 >2014-04-16 07:41:21,619 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 08:36:04,389 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-04-16 08:36:05,197 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-04-16 08:36:05,198 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Lok >2014-04-16 08:36:05,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Lok >2014-04-16 08:36:05,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Lok >2014-04-16 08:36:05,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center Lok >2014-04-16 08:36:05,199 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 08:36:05,200 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 08:41:21,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6f926688 >2014-04-16 08:41:21,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7489a3f1 >2014-04-16 08:41:21,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7489a3f1 >2014-04-16 08:41:21,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6f926688 >2014-04-16 08:41:21,862 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 09:36:04,391 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-04-16 09:36:05,200 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-04-16 09:36:05,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-04-16 09:36:05,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-04-16 09:36:05,201 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-04-16 09:36:05,202 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-04-16 09:36:05,202 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 09:36:05,202 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 09:41:21,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 629fd2cb >2014-04-16 09:41:21,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6ac84002 >2014-04-16 09:41:22,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ac84002 >2014-04-16 09:41:22,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 629fd2cb >2014-04-16 09:41:22,096 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 10:20:25,491 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-6) Running command: LoginAdminUserCommand internal: false. >2014-04-16 10:20:25,505 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-16 10:20:25,569 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginUserCommand internal: false. >2014-04-16 10:20:25,580 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-16 10:20:42,562 INFO [org.ovirt.engine.core.bll.SetVmTicketCommand] (ajp-/127.0.0.1:8702-4) [1e3e1fd] Running command: SetVmTicketCommand internal: false. Entities affected : ID: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 Type: VM >2014-04-16 10:20:42,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [1e3e1fd] START, SetVmTicketVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, ticket=AE+OVBeFI1Bs, validTime=120,m userName=admin@internal, userId=fdfc627c-d875-11e0-90f0-83df133b58cc), log id: 545d8a4e >2014-04-16 10:20:42,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetVmTicketVDSCommand] (ajp-/127.0.0.1:8702-4) [1e3e1fd] FINISH, SetVmTicketVDSCommand, log id: 545d8a4e >2014-04-16 10:20:42,878 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [1e3e1fd] Correlation ID: 1e3e1fd, Call Stack: null, Custom Event ID: -1, Message: user admin@internal initiated console session for VM Gal >2014-04-16 10:21:53,226 INFO [org.ovirt.engine.core.bll.StopVmCommand] (pool-4-thread-48) [76f422be] Running command: StopVmCommand internal: false. Entities affected : ID: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 Type: VM >2014-04-16 10:21:53,236 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (pool-4-thread-48) [76f422be] START, DestroyVmVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, force=false, secondsToWait=0, gracefully=false), log id: 21b4e96f >2014-04-16 10:21:53,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (pool-4-thread-48) [76f422be] START, DestroyVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vmId=b066ff4b-970f-4305-94a4-1f32f7fa6fe2, force=false, secondsToWait=0, gracefully=false), log id: 2d354fc1 >2014-04-16 10:21:54,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DestroyVDSCommand] (pool-4-thread-48) [76f422be] FINISH, DestroyVDSCommand, log id: 2d354fc1 >2014-04-16 10:21:54,172 INFO [org.ovirt.engine.core.vdsbroker.DestroyVmVDSCommand] (pool-4-thread-48) [76f422be] FINISH, DestroyVmVDSCommand, return: Down, log id: 21b4e96f >2014-04-16 10:21:54,194 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-48) [76f422be] Correlation ID: 76f422be, Job ID: b6b03697-5692-4184-b609-43bbd7c9f293, Call Stack: null, Custom Event ID: -1, Message: VM Gal powered off by admin@internal (Host: 10.34.27.209). >2014-04-16 10:36:04,393 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-04-16 10:36:05,202 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-04-16 10:36:05,203 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VM OVFs in Data Center Lok >2014-04-16 10:36:05,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated VM OVFs in Data Center Lok >2014-04-16 10:36:05,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update template OVFs in Data Center Lok >2014-04-16 10:36:05,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated templates OVFs in Data Center Lok >2014-04-16 10:36:05,204 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 10:36:05,205 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 10:41:22,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7abb25c >2014-04-16 10:41:22,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 194ab9f3 >2014-04-16 10:41:22,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 194ab9f3 >2014-04-16 10:41:22,339 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7abb25c >2014-04-16 10:41:22,343 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 11:36:04,394 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-20) Start refreshing all users data >2014-04-16 11:36:05,205 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-04-16 11:36:05,205 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VM OVFs in Data Center Lok >2014-04-16 11:36:05,206 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated VM OVFs in Data Center Lok >2014-04-16 11:36:05,206 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update template OVFs in Data Center Lok >2014-04-16 11:36:05,207 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated templates OVFs in Data Center Lok >2014-04-16 11:36:05,207 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 11:36:05,207 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 11:41:22,347 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 63078791 >2014-04-16 11:41:22,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3557362 >2014-04-16 11:41:22,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3557362 >2014-04-16 11:41:22,575 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 63078791 >2014-04-16 11:41:22,580 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 12:36:04,396 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-04-16 12:36:05,208 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-16 12:36:05,209 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-16 12:36:05,209 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-16 12:36:05,209 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-16 12:36:05,210 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-16 12:36:05,210 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 12:36:05,210 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 12:41:22,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6160cbcf >2014-04-16 12:41:22,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 19e4b4fb >2014-04-16 12:41:22,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 19e4b4fb >2014-04-16 12:41:22,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6160cbcf >2014-04-16 12:41:22,813 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 13:36:04,398 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-04-16 13:36:05,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-04-16 13:36:05,213 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VM OVFs in Data Center Lok >2014-04-16 13:36:05,213 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated VM OVFs in Data Center Lok >2014-04-16 13:36:05,213 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update template OVFs in Data Center Lok >2014-04-16 13:36:05,214 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated templates OVFs in Data Center Lok >2014-04-16 13:36:05,214 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 13:36:05,214 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 13:41:22,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 64de30e8 >2014-04-16 13:41:22,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 375d0314 >2014-04-16 13:41:23,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 375d0314 >2014-04-16 13:41:23,366 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 64de30e8 >2014-04-16 13:41:23,374 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 14:36:04,399 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-04-16 14:36:05,214 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-04-16 14:36:05,215 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VM OVFs in Data Center Lok >2014-04-16 14:36:05,216 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated VM OVFs in Data Center Lok >2014-04-16 14:36:05,216 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update template OVFs in Data Center Lok >2014-04-16 14:36:05,216 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated templates OVFs in Data Center Lok >2014-04-16 14:36:05,216 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 14:36:05,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 14:41:23,378 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 26d84171 >2014-04-16 14:41:23,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 59fb40ff >2014-04-16 14:41:23,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 59fb40ff >2014-04-16 14:41:23,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 26d84171 >2014-04-16 14:41:23,631 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 15:36:04,401 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-04-16 15:36:05,217 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-16 15:36:05,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-16 15:36:05,219 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-16 15:36:05,219 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-16 15:36:05,219 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-16 15:36:05,219 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 15:36:05,220 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 15:41:23,634 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7b88df06 >2014-04-16 15:41:23,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 56ea4246 >2014-04-16 15:41:23,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 56ea4246 >2014-04-16 15:41:23,855 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7b88df06 >2014-04-16 15:41:23,860 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 16:36:04,403 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Start refreshing all users data >2014-04-16 16:36:05,220 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-04-16 16:36:05,221 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VM OVFs in Data Center Lok >2014-04-16 16:36:05,222 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated VM OVFs in Data Center Lok >2014-04-16 16:36:05,222 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update template OVFs in Data Center Lok >2014-04-16 16:36:05,223 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated templates OVFs in Data Center Lok >2014-04-16 16:36:05,223 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 16:36:05,223 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 16:41:23,864 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 68bf1bce >2014-04-16 16:41:23,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5abaa71b >2014-04-16 16:41:24,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5abaa71b >2014-04-16 16:41:24,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 68bf1bce >2014-04-16 16:41:24,393 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 17:36:04,403 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) Start refreshing all users data >2014-04-16 17:36:05,223 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-04-16 17:36:05,224 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VM OVFs in Data Center Lok >2014-04-16 17:36:05,224 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated VM OVFs in Data Center Lok >2014-04-16 17:36:05,224 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update template OVFs in Data Center Lok >2014-04-16 17:36:05,225 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated templates OVFs in Data Center Lok >2014-04-16 17:36:05,225 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 17:36:05,226 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 17:41:24,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7f54455 >2014-04-16 17:41:24,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 77592c87 >2014-04-16 17:41:24,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 77592c87 >2014-04-16 17:41:24,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f54455 >2014-04-16 17:41:24,815 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 18:36:04,405 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-04-16 18:36:05,227 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-04-16 18:36:05,228 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VM OVFs in Data Center Lok >2014-04-16 18:36:05,228 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated VM OVFs in Data Center Lok >2014-04-16 18:36:05,228 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update template OVFs in Data Center Lok >2014-04-16 18:36:05,229 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated templates OVFs in Data Center Lok >2014-04-16 18:36:05,229 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 18:36:05,229 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 18:41:24,820 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 61301f4 >2014-04-16 18:41:24,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 49e53d49 >2014-04-16 18:41:25,040 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 49e53d49 >2014-04-16 18:41:25,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 61301f4 >2014-04-16 18:41:25,046 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 19:36:04,407 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-04-16 19:36:05,227 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-04-16 19:36:05,228 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VM OVFs in Data Center Lok >2014-04-16 19:36:05,229 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated VM OVFs in Data Center Lok >2014-04-16 19:36:05,229 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update template OVFs in Data Center Lok >2014-04-16 19:36:05,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated templates OVFs in Data Center Lok >2014-04-16 19:36:05,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 19:36:05,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 19:41:25,049 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 9ee7ae1 >2014-04-16 19:41:25,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7c54edf >2014-04-16 19:41:25,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c54edf >2014-04-16 19:41:25,274 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 9ee7ae1 >2014-04-16 19:41:25,279 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 20:36:04,409 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-04-16 20:36:05,231 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-04-16 20:36:05,232 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VM OVFs in Data Center Lok >2014-04-16 20:36:05,232 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated VM OVFs in Data Center Lok >2014-04-16 20:36:05,232 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update template OVFs in Data Center Lok >2014-04-16 20:36:05,233 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated templates OVFs in Data Center Lok >2014-04-16 20:36:05,233 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 20:36:05,234 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 20:41:25,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 119b31a2 >2014-04-16 20:41:25,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 63d78bb5 >2014-04-16 20:41:25,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 63d78bb5 >2014-04-16 20:41:25,509 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 119b31a2 >2014-04-16 20:41:25,514 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 21:36:04,411 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-16 21:36:05,234 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-16 21:36:05,235 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-04-16 21:36:05,236 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-04-16 21:36:05,236 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-04-16 21:36:05,237 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-04-16 21:36:05,237 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 21:36:05,237 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 21:41:25,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4bc2608d >2014-04-16 21:41:25,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4650efd6 >2014-04-16 21:41:25,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4650efd6 >2014-04-16 21:41:25,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4bc2608d >2014-04-16 21:41:25,748 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 22:36:04,414 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-04-16 22:36:05,237 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-04-16 22:36:05,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VM OVFs in Data Center Lok >2014-04-16 22:36:05,239 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated VM OVFs in Data Center Lok >2014-04-16 22:36:05,239 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update template OVFs in Data Center Lok >2014-04-16 22:36:05,239 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated templates OVFs in Data Center Lok >2014-04-16 22:36:05,239 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 22:36:05,240 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 22:41:25,752 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3b815473 >2014-04-16 22:41:25,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 27cf688a >2014-04-16 22:41:26,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 27cf688a >2014-04-16 22:41:26,158 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3b815473 >2014-04-16 22:41:26,163 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-16 23:36:04,415 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Start refreshing all users data >2014-04-16 23:36:05,240 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-04-16 23:36:05,241 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VM OVFs in Data Center Lok >2014-04-16 23:36:05,242 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully updated VM OVFs in Data Center Lok >2014-04-16 23:36:05,242 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update template OVFs in Data Center Lok >2014-04-16 23:36:05,242 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully updated templates OVFs in Data Center Lok >2014-04-16 23:36:05,242 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-16 23:36:05,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-16 23:41:26,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1f87d74b >2014-04-16 23:41:26,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: f478d19 >2014-04-16 23:41:26,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: f478d19 >2014-04-16 23:41:26,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1f87d74b >2014-04-16 23:41:26,563 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 00:36:04,417 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-04-17 00:36:05,243 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-17 00:36:05,244 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-17 00:36:05,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-17 00:36:05,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-17 00:36:05,247 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-17 00:36:05,247 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 00:36:05,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 00:41:26,565 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7861eaf7 >2014-04-17 00:41:26,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 64893bdf >2014-04-17 00:41:26,798 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 64893bdf >2014-04-17 00:41:26,798 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7861eaf7 >2014-04-17 00:41:26,802 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 01:36:04,419 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-17 01:36:05,249 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-04-17 01:36:05,249 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-04-17 01:36:05,250 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-04-17 01:36:05,250 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-04-17 01:36:05,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-04-17 01:36:05,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 01:36:05,252 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 01:41:26,806 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6c43db7a >2014-04-17 01:41:26,806 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5afe5e1d >2014-04-17 01:41:27,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5afe5e1d >2014-04-17 01:41:27,027 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6c43db7a >2014-04-17 01:41:27,032 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 02:36:04,421 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-17 02:36:05,252 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-17 02:36:05,253 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-17 02:36:05,254 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-17 02:36:05,254 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-17 02:36:05,255 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-17 02:36:05,255 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 02:36:05,255 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 02:41:27,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 34176cd3 >2014-04-17 02:41:27,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6de9ab5c >2014-04-17 02:41:27,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6de9ab5c >2014-04-17 02:41:27,256 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 34176cd3 >2014-04-17 02:41:27,261 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-57) Start deleteAgedOutAuditLogs >2014-04-17 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-57) Finished deleteAgedOutAuditLogs >2014-04-17 03:36:04,423 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-5) Start refreshing all users data >2014-04-17 03:36:05,256 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-04-17 03:36:05,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VM OVFs in Data Center Lok >2014-04-17 03:36:05,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated VM OVFs in Data Center Lok >2014-04-17 03:36:05,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update template OVFs in Data Center Lok >2014-04-17 03:36:05,258 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated templates OVFs in Data Center Lok >2014-04-17 03:36:05,258 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 03:36:05,259 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 03:41:27,265 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6b89ca14 >2014-04-17 03:41:27,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 23321e34 >2014-04-17 03:41:27,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23321e34 >2014-04-17 03:41:27,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6b89ca14 >2014-04-17 03:41:27,800 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 04:36:04,426 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-04-17 04:36:05,259 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-04-17 04:36:05,260 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VM OVFs in Data Center Lok >2014-04-17 04:36:05,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully updated VM OVFs in Data Center Lok >2014-04-17 04:36:05,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update template OVFs in Data Center Lok >2014-04-17 04:36:05,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully updated templates OVFs in Data Center Lok >2014-04-17 04:36:05,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 04:36:05,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 04:41:27,804 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 76583a97 >2014-04-17 04:41:27,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1a6ecc34 >2014-04-17 04:41:28,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1a6ecc34 >2014-04-17 04:41:28,029 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76583a97 >2014-04-17 04:41:28,033 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 05:36:04,426 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-21) Start refreshing all users data >2014-04-17 05:36:05,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-04-17 05:36:05,264 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VM OVFs in Data Center Lok >2014-04-17 05:36:05,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated VM OVFs in Data Center Lok >2014-04-17 05:36:05,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update template OVFs in Data Center Lok >2014-04-17 05:36:05,266 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated templates OVFs in Data Center Lok >2014-04-17 05:36:05,266 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 05:36:05,266 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 05:41:28,037 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1914070e >2014-04-17 05:41:28,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 71db34fe >2014-04-17 05:41:28,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 71db34fe >2014-04-17 05:41:28,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1914070e >2014-04-17 05:41:28,487 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 06:36:04,428 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-04-17 06:36:05,267 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-04-17 06:36:05,267 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VM OVFs in Data Center Lok >2014-04-17 06:36:05,268 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated VM OVFs in Data Center Lok >2014-04-17 06:36:05,268 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update template OVFs in Data Center Lok >2014-04-17 06:36:05,269 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated templates OVFs in Data Center Lok >2014-04-17 06:36:05,269 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 06:36:05,269 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 06:41:28,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2daf14f0 >2014-04-17 06:41:28,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4187affe >2014-04-17 06:41:28,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4187affe >2014-04-17 06:41:28,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2daf14f0 >2014-04-17 06:41:28,718 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 07:36:04,429 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-72) Start refreshing all users data >2014-04-17 07:36:05,269 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-04-17 07:36:05,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VM OVFs in Data Center Lok >2014-04-17 07:36:05,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated VM OVFs in Data Center Lok >2014-04-17 07:36:05,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update template OVFs in Data Center Lok >2014-04-17 07:36:05,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated templates OVFs in Data Center Lok >2014-04-17 07:36:05,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 07:36:05,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 07:41:28,723 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 10d14d45 >2014-04-17 07:41:28,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1d27f5a3 >2014-04-17 07:41:28,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d27f5a3 >2014-04-17 07:41:28,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 10d14d45 >2014-04-17 07:41:28,954 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 08:36:04,432 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-04-17 08:36:05,272 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-04-17 08:36:05,273 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VM OVFs in Data Center Lok >2014-04-17 08:36:05,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated VM OVFs in Data Center Lok >2014-04-17 08:36:05,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update template OVFs in Data Center Lok >2014-04-17 08:36:05,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated templates OVFs in Data Center Lok >2014-04-17 08:36:05,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 08:36:05,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 08:41:28,958 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 222a8541 >2014-04-17 08:41:28,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 15927fed >2014-04-17 08:41:29,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 15927fed >2014-04-17 08:41:29,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 222a8541 >2014-04-17 08:41:29,189 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 09:36:04,435 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-04-17 09:36:05,276 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-04-17 09:36:05,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VM OVFs in Data Center Lok >2014-04-17 09:36:05,278 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated VM OVFs in Data Center Lok >2014-04-17 09:36:05,278 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update template OVFs in Data Center Lok >2014-04-17 09:36:05,279 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated templates OVFs in Data Center Lok >2014-04-17 09:36:05,279 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 09:36:05,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 09:41:29,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4985863a >2014-04-17 09:41:29,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5d2f6f8b >2014-04-17 09:41:29,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5d2f6f8b >2014-04-17 09:41:29,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4985863a >2014-04-17 09:41:29,425 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 10:36:04,437 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-76) Start refreshing all users data >2014-04-17 10:36:05,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-04-17 10:36:05,281 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VM OVFs in Data Center Lok >2014-04-17 10:36:05,282 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated VM OVFs in Data Center Lok >2014-04-17 10:36:05,282 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update template OVFs in Data Center Lok >2014-04-17 10:36:05,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated templates OVFs in Data Center Lok >2014-04-17 10:36:05,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 10:36:05,284 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 10:41:29,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 19c4971e >2014-04-17 10:41:29,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 399d3fc9 >2014-04-17 10:41:29,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 399d3fc9 >2014-04-17 10:41:29,660 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 19c4971e >2014-04-17 10:41:29,665 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 11:36:04,440 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-04-17 11:36:05,285 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-04-17 11:36:05,286 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VM OVFs in Data Center Lok >2014-04-17 11:36:05,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated VM OVFs in Data Center Lok >2014-04-17 11:36:05,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update template OVFs in Data Center Lok >2014-04-17 11:36:05,288 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated templates OVFs in Data Center Lok >2014-04-17 11:36:05,288 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 11:36:05,288 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 11:41:29,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 13cf026f >2014-04-17 11:41:29,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7ba8cf91 >2014-04-17 11:41:29,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7ba8cf91 >2014-04-17 11:41:29,897 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 13cf026f >2014-04-17 11:41:29,902 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 12:36:04,442 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-04-17 12:36:05,289 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-04-17 12:36:05,289 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VM OVFs in Data Center Lok >2014-04-17 12:36:05,290 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated VM OVFs in Data Center Lok >2014-04-17 12:36:05,290 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update template OVFs in Data Center Lok >2014-04-17 12:36:05,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated templates OVFs in Data Center Lok >2014-04-17 12:36:05,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 12:36:05,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 12:41:29,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 51bc7005 >2014-04-17 12:41:29,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2a5e822f >2014-04-17 12:41:30,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2a5e822f >2014-04-17 12:41:30,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 51bc7005 >2014-04-17 12:41:30,322 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 13:36:04,444 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-04-17 13:36:05,292 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-04-17 13:36:05,293 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VM OVFs in Data Center Lok >2014-04-17 13:36:05,293 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated VM OVFs in Data Center Lok >2014-04-17 13:36:05,293 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update template OVFs in Data Center Lok >2014-04-17 13:36:05,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated templates OVFs in Data Center Lok >2014-04-17 13:36:05,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 13:36:05,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 13:41:30,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: a425cc5 >2014-04-17 13:41:30,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: ad88ec6 >2014-04-17 13:41:30,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ad88ec6 >2014-04-17 13:41:30,558 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: a425cc5 >2014-04-17 13:41:30,563 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 14:36:04,446 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-04-17 14:36:05,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-04-17 14:36:05,295 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VM OVFs in Data Center Lok >2014-04-17 14:36:05,296 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated VM OVFs in Data Center Lok >2014-04-17 14:36:05,296 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update template OVFs in Data Center Lok >2014-04-17 14:36:05,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated templates OVFs in Data Center Lok >2014-04-17 14:36:05,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 14:36:05,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 14:41:30,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7f1f5fb2 >2014-04-17 14:41:30,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 64c3e051 >2014-04-17 14:41:30,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 64c3e051 >2014-04-17 14:41:30,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f1f5fb2 >2014-04-17 14:41:30,805 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 15:36:04,447 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-04-17 15:36:05,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-17 15:36:05,298 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-17 15:36:05,299 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-17 15:36:05,299 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-17 15:36:05,301 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-17 15:36:05,301 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 15:36:05,301 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 15:41:30,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 639199f5 >2014-04-17 15:41:30,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6c12d6e4 >2014-04-17 15:41:31,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6c12d6e4 >2014-04-17 15:41:31,036 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 639199f5 >2014-04-17 15:41:31,041 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 16:36:04,449 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-17 16:36:05,301 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-17 16:36:05,302 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-04-17 16:36:05,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-04-17 16:36:05,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-04-17 16:36:05,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-04-17 16:36:05,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 16:36:05,304 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 16:41:31,045 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7c9edde7 >2014-04-17 16:41:31,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 21faa89c >2014-04-17 16:41:31,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 21faa89c >2014-04-17 16:41:31,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c9edde7 >2014-04-17 16:41:31,284 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 17:36:04,451 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-79) Start refreshing all users data >2014-04-17 17:36:05,304 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-04-17 17:36:05,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VM OVFs in Data Center Lok >2014-04-17 17:36:05,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated VM OVFs in Data Center Lok >2014-04-17 17:36:05,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update template OVFs in Data Center Lok >2014-04-17 17:36:05,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated templates OVFs in Data Center Lok >2014-04-17 17:36:05,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 17:36:05,307 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 17:41:31,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2e881b8a >2014-04-17 17:41:31,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 69c818e3 >2014-04-17 17:41:31,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 69c818e3 >2014-04-17 17:41:31,512 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e881b8a >2014-04-17 17:41:31,516 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 18:36:04,453 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-04-17 18:36:05,308 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-04-17 18:36:05,308 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-04-17 18:36:05,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-04-17 18:36:05,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-04-17 18:36:05,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-04-17 18:36:05,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 18:36:05,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 18:41:31,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5a97e78c >2014-04-17 18:41:31,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2ae2c6cb >2014-04-17 18:41:31,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2ae2c6cb >2014-04-17 18:41:31,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a97e78c >2014-04-17 18:41:31,752 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 19:36:04,455 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-04-17 19:36:05,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-04-17 19:36:05,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VM OVFs in Data Center Lok >2014-04-17 19:36:05,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated VM OVFs in Data Center Lok >2014-04-17 19:36:05,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update template OVFs in Data Center Lok >2014-04-17 19:36:05,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated templates OVFs in Data Center Lok >2014-04-17 19:36:05,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 19:36:05,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 19:41:31,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5624a2a9 >2014-04-17 19:41:31,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7f33316d >2014-04-17 19:41:31,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7f33316d >2014-04-17 19:41:31,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5624a2a9 >2014-04-17 19:41:31,989 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 20:36:04,455 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-04-17 20:36:05,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-17 20:36:05,314 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-04-17 20:36:05,315 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-04-17 20:36:05,315 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-04-17 20:36:05,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-04-17 20:36:05,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 20:36:05,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 20:41:31,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 31fb22ab >2014-04-17 20:41:31,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5ea1222f >2014-04-17 20:41:32,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5ea1222f >2014-04-17 20:41:32,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 31fb22ab >2014-04-17 20:41:32,222 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 21:36:04,457 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-04-17 21:36:05,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-04-17 21:36:05,317 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VM OVFs in Data Center Lok >2014-04-17 21:36:05,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated VM OVFs in Data Center Lok >2014-04-17 21:36:05,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update template OVFs in Data Center Lok >2014-04-17 21:36:05,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated templates OVFs in Data Center Lok >2014-04-17 21:36:05,318 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 21:36:05,319 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 21:41:32,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 20deeb71 >2014-04-17 21:41:32,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: b335893 >2014-04-17 21:41:32,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: b335893 >2014-04-17 21:41:32,448 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 20deeb71 >2014-04-17 21:41:32,453 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 22:36:04,459 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-04-17 22:36:05,320 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-17 22:36:05,320 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-17 22:36:05,321 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-17 22:36:05,321 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-17 22:36:05,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-17 22:36:05,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 22:36:05,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 22:41:32,458 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1772bf87 >2014-04-17 22:41:32,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6e7c4417 >2014-04-17 22:41:32,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6e7c4417 >2014-04-17 22:41:32,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1772bf87 >2014-04-17 22:41:32,686 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-17 23:36:04,461 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-04-17 23:36:05,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-17 23:36:05,323 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-17 23:36:05,324 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-17 23:36:05,324 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-17 23:36:05,324 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-17 23:36:05,324 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-17 23:36:05,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-17 23:41:32,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 60798d69 >2014-04-17 23:41:32,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4c3319fa >2014-04-17 23:41:32,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4c3319fa >2014-04-17 23:41:32,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 60798d69 >2014-04-17 23:41:32,918 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 00:36:04,462 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-04-18 00:36:05,325 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-04-18 00:36:05,326 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VM OVFs in Data Center Lok >2014-04-18 00:36:05,326 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully updated VM OVFs in Data Center Lok >2014-04-18 00:36:05,327 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update template OVFs in Data Center Lok >2014-04-18 00:36:05,327 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully updated templates OVFs in Data Center Lok >2014-04-18 00:36:05,327 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 00:36:05,327 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 00:41:32,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7b982edc >2014-04-18 00:41:32,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 442cace8 >2014-04-18 00:41:33,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 442cace8 >2014-04-18 00:41:33,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7b982edc >2014-04-18 00:41:33,153 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 01:36:04,463 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-89) Start refreshing all users data >2014-04-18 01:36:05,328 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-04-18 01:36:05,329 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VM OVFs in Data Center Lok >2014-04-18 01:36:05,331 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated VM OVFs in Data Center Lok >2014-04-18 01:36:05,331 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update template OVFs in Data Center Lok >2014-04-18 01:36:05,332 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated templates OVFs in Data Center Lok >2014-04-18 01:36:05,332 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 01:36:05,332 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 01:41:33,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6871ea83 >2014-04-18 01:41:33,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: edb7407 >2014-04-18 01:41:33,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: edb7407 >2014-04-18 01:41:33,375 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6871ea83 >2014-04-18 01:41:33,379 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 02:36:04,465 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-04-18 02:36:05,332 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VMs/Templates Ovf. >2014-04-18 02:36:05,333 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update VM OVFs in Data Center Lok >2014-04-18 02:36:05,333 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated VM OVFs in Data Center Lok >2014-04-18 02:36:05,334 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to update template OVFs in Data Center Lok >2014-04-18 02:36:05,334 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully updated templates OVFs in Data Center Lok >2014-04-18 02:36:05,334 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 02:36:05,335 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 02:41:33,383 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 61c68002 >2014-04-18 02:41:33,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2efe3574 >2014-04-18 02:41:33,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2efe3574 >2014-04-18 02:41:33,603 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 61c68002 >2014-04-18 02:41:33,608 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-100) Start deleteAgedOutAuditLogs >2014-04-18 03:35:35,004 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-100) Finished deleteAgedOutAuditLogs >2014-04-18 03:36:04,466 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-18 03:36:05,335 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-18 03:36:05,336 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-18 03:36:05,342 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-18 03:36:05,342 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-18 03:36:05,343 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-18 03:36:05,343 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 03:36:05,343 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 03:41:33,612 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3941e952 >2014-04-18 03:41:33,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 163d23dc >2014-04-18 03:41:33,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 163d23dc >2014-04-18 03:41:33,839 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3941e952 >2014-04-18 03:41:33,844 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 04:36:04,468 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-04-18 04:36:05,344 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-04-18 04:36:05,345 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VM OVFs in Data Center Lok >2014-04-18 04:36:05,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated VM OVFs in Data Center Lok >2014-04-18 04:36:05,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update template OVFs in Data Center Lok >2014-04-18 04:36:05,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated templates OVFs in Data Center Lok >2014-04-18 04:36:05,346 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 04:36:05,347 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 04:41:33,847 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [6d63ed15] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 28c95bd9 >2014-04-18 04:41:33,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [6d63ed15] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3bad187a >2014-04-18 04:41:34,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [6d63ed15] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3bad187a >2014-04-18 04:41:34,421 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [6d63ed15] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 28c95bd9 >2014-04-18 04:41:34,425 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) [6d63ed15] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 05:36:04,470 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-04-18 05:36:05,347 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-04-18 05:36:05,348 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VM OVFs in Data Center Lok >2014-04-18 05:36:05,349 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated VM OVFs in Data Center Lok >2014-04-18 05:36:05,349 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update template OVFs in Data Center Lok >2014-04-18 05:36:05,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated templates OVFs in Data Center Lok >2014-04-18 05:36:05,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 05:36:05,350 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 05:41:34,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 32f7fc94 >2014-04-18 05:41:34,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 36e772b0 >2014-04-18 05:41:34,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 36e772b0 >2014-04-18 05:41:34,652 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 32f7fc94 >2014-04-18 05:41:34,676 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 06:36:04,473 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-76) Start refreshing all users data >2014-04-18 06:36:05,351 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-04-18 06:36:05,351 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VM OVFs in Data Center Lok >2014-04-18 06:36:05,352 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated VM OVFs in Data Center Lok >2014-04-18 06:36:05,352 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update template OVFs in Data Center Lok >2014-04-18 06:36:05,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated templates OVFs in Data Center Lok >2014-04-18 06:36:05,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 06:36:05,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 06:41:34,680 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: ac092c3 >2014-04-18 06:41:34,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 68e345b8 >2014-04-18 06:41:34,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 68e345b8 >2014-04-18 06:41:34,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ac092c3 >2014-04-18 06:41:34,902 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 07:36:04,475 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-50) Start refreshing all users data >2014-04-18 07:36:05,353 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-04-18 07:36:05,354 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VM OVFs in Data Center Lok >2014-04-18 07:36:05,355 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated VM OVFs in Data Center Lok >2014-04-18 07:36:05,355 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update template OVFs in Data Center Lok >2014-04-18 07:36:05,356 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated templates OVFs in Data Center Lok >2014-04-18 07:36:05,356 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 07:36:05,356 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 07:41:34,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 199d68fa >2014-04-18 07:41:34,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2d570239 >2014-04-18 07:41:35,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d570239 >2014-04-18 07:41:35,129 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 199d68fa >2014-04-18 07:41:35,134 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 08:36:04,477 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-54) Start refreshing all users data >2014-04-18 08:36:05,357 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-04-18 08:36:05,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VM OVFs in Data Center Lok >2014-04-18 08:36:05,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated VM OVFs in Data Center Lok >2014-04-18 08:36:05,358 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update template OVFs in Data Center Lok >2014-04-18 08:36:05,360 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated templates OVFs in Data Center Lok >2014-04-18 08:36:05,360 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 08:36:05,361 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 08:41:35,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6f52d6ac >2014-04-18 08:41:35,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 278bef4d >2014-04-18 08:41:35,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 278bef4d >2014-04-18 08:41:35,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6f52d6ac >2014-04-18 08:41:35,556 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 09:36:04,478 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-87) Start refreshing all users data >2014-04-18 09:36:05,361 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-04-18 09:36:05,362 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VM OVFs in Data Center Lok >2014-04-18 09:36:05,363 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated VM OVFs in Data Center Lok >2014-04-18 09:36:05,363 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update template OVFs in Data Center Lok >2014-04-18 09:36:05,364 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated templates OVFs in Data Center Lok >2014-04-18 09:36:05,364 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 09:36:05,365 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 09:41:35,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 23547d8 >2014-04-18 09:41:35,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 77ffb709 >2014-04-18 09:41:35,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 77ffb709 >2014-04-18 09:41:35,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23547d8 >2014-04-18 09:41:35,798 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 10:36:04,478 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-04-18 10:36:05,366 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VMs/Templates Ovf. >2014-04-18 10:36:05,367 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update VM OVFs in Data Center Lok >2014-04-18 10:36:05,369 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Successfully updated VM OVFs in Data Center Lok >2014-04-18 10:36:05,369 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to update template OVFs in Data Center Lok >2014-04-18 10:36:05,369 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Successfully updated templates OVFs in Data Center Lok >2014-04-18 10:36:05,370 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 10:36:05,370 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 10:41:35,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 24561970 >2014-04-18 10:41:35,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 10a01ab3 >2014-04-18 10:41:36,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 10a01ab3 >2014-04-18 10:41:36,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 24561970 >2014-04-18 10:41:36,031 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 11:36:04,480 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-04-18 11:36:05,370 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-04-18 11:36:05,371 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Lok >2014-04-18 11:36:05,372 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Lok >2014-04-18 11:36:05,372 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Lok >2014-04-18 11:36:05,373 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated templates OVFs in Data Center Lok >2014-04-18 11:36:05,373 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 11:36:05,373 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 11:41:36,035 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4f22c732 >2014-04-18 11:41:36,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7443cc34 >2014-04-18 11:41:36,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7443cc34 >2014-04-18 11:41:36,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4f22c732 >2014-04-18 11:41:36,284 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 12:36:04,481 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-04-18 12:36:05,373 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-18 12:36:05,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-18 12:36:05,374 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-18 12:36:05,375 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-18 12:36:05,375 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-18 12:36:05,375 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 12:36:05,376 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 12:41:36,289 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 336696a4 >2014-04-18 12:41:36,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 142c0359 >2014-04-18 12:41:36,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 142c0359 >2014-04-18 12:41:36,513 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 336696a4 >2014-04-18 12:41:36,518 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 13:36:04,482 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-49) Start refreshing all users data >2014-04-18 13:36:05,376 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-04-18 13:36:05,377 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VM OVFs in Data Center Lok >2014-04-18 13:36:05,378 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated VM OVFs in Data Center Lok >2014-04-18 13:36:05,378 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update template OVFs in Data Center Lok >2014-04-18 13:36:05,378 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated templates OVFs in Data Center Lok >2014-04-18 13:36:05,378 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 13:36:05,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 13:41:36,523 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 71f9216a >2014-04-18 13:41:36,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1b00f09b >2014-04-18 13:41:36,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b00f09b >2014-04-18 13:41:36,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 71f9216a >2014-04-18 13:41:36,768 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 14:36:04,484 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-18 14:36:05,379 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-04-18 14:36:05,380 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-04-18 14:36:05,381 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-04-18 14:36:05,381 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-04-18 14:36:05,381 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-04-18 14:36:05,381 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 14:36:05,382 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 14:41:36,770 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 63eabcbb >2014-04-18 14:41:36,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 480ccc3c >2014-04-18 14:41:36,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 480ccc3c >2014-04-18 14:41:36,997 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 63eabcbb >2014-04-18 14:41:37,002 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 15:36:04,485 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) Start refreshing all users data >2014-04-18 15:36:05,382 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-18 15:36:05,384 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-18 15:36:05,384 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-18 15:36:05,384 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-18 15:36:05,385 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-18 15:36:05,385 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 15:36:05,386 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 15:41:37,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 619408a2 >2014-04-18 15:41:37,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3dc2f5ba >2014-04-18 15:41:37,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3dc2f5ba >2014-04-18 15:41:37,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 619408a2 >2014-04-18 15:41:37,236 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 16:36:04,488 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-04-18 16:36:05,386 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update VMs/Templates Ovf. >2014-04-18 16:36:05,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update VM OVFs in Data Center Lok >2014-04-18 16:36:05,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully updated VM OVFs in Data Center Lok >2014-04-18 16:36:05,387 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update template OVFs in Data Center Lok >2014-04-18 16:36:05,388 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully updated templates OVFs in Data Center Lok >2014-04-18 16:36:05,388 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 16:36:05,388 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 16:41:37,239 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 12c2b929 >2014-04-18 16:41:37,240 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2e0a3c5e >2014-04-18 16:41:37,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e0a3c5e >2014-04-18 16:41:37,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 12c2b929 >2014-04-18 16:41:37,467 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 17:36:04,490 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-04-18 17:36:05,388 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-04-18 17:36:05,389 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VM OVFs in Data Center Lok >2014-04-18 17:36:05,390 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated VM OVFs in Data Center Lok >2014-04-18 17:36:05,390 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update template OVFs in Data Center Lok >2014-04-18 17:36:05,390 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated templates OVFs in Data Center Lok >2014-04-18 17:36:05,390 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 17:36:05,391 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 17:41:37,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7d9cc83a >2014-04-18 17:41:37,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 401d9843 >2014-04-18 17:41:37,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 401d9843 >2014-04-18 17:41:37,701 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7d9cc83a >2014-04-18 17:41:37,706 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 18:36:04,492 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-04-18 18:36:05,391 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-04-18 18:36:05,392 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VM OVFs in Data Center Lok >2014-04-18 18:36:05,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated VM OVFs in Data Center Lok >2014-04-18 18:36:05,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update template OVFs in Data Center Lok >2014-04-18 18:36:05,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated templates OVFs in Data Center Lok >2014-04-18 18:36:05,393 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 18:36:05,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 18:41:37,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 15547a58 >2014-04-18 18:41:37,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5d2c3258 >2014-04-18 18:41:37,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5d2c3258 >2014-04-18 18:41:37,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 15547a58 >2014-04-18 18:41:37,938 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 19:36:04,495 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-43) Start refreshing all users data >2014-04-18 19:36:05,394 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-04-18 19:36:05,395 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VM OVFs in Data Center Lok >2014-04-18 19:36:05,396 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated VM OVFs in Data Center Lok >2014-04-18 19:36:05,396 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update template OVFs in Data Center Lok >2014-04-18 19:36:05,396 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated templates OVFs in Data Center Lok >2014-04-18 19:36:05,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 19:36:05,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 19:41:37,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 23f4e3ef >2014-04-18 19:41:37,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 22a0725a >2014-04-18 19:41:38,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 22a0725a >2014-04-18 19:41:38,168 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23f4e3ef >2014-04-18 19:41:38,173 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 20:36:04,496 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-04-18 20:36:05,397 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-18 20:36:05,399 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-18 20:36:05,400 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-18 20:36:05,400 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-18 20:36:05,400 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-18 20:36:05,400 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 20:36:05,401 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 20:41:38,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 20b618a9 >2014-04-18 20:41:38,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 130d1006 >2014-04-18 20:41:38,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 130d1006 >2014-04-18 20:41:38,404 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 20b618a9 >2014-04-18 20:41:38,410 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 21:36:04,498 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-04-18 21:36:05,402 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Attempting to update VMs/Templates Ovf. >2014-04-18 21:36:05,402 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Attempting to update VM OVFs in Data Center Lok >2014-04-18 21:36:05,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Successfully updated VM OVFs in Data Center Lok >2014-04-18 21:36:05,403 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Attempting to update template OVFs in Data Center Lok >2014-04-18 21:36:05,404 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Successfully updated templates OVFs in Data Center Lok >2014-04-18 21:36:05,404 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 21:36:05,404 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 21:41:38,411 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6cca360e >2014-04-18 21:41:38,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4fa308ee >2014-04-18 21:41:38,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4fa308ee >2014-04-18 21:41:38,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6cca360e >2014-04-18 21:41:38,643 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 22:36:04,499 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-66) Start refreshing all users data >2014-04-18 22:36:05,405 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-04-18 22:36:05,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VM OVFs in Data Center Lok >2014-04-18 22:36:05,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated VM OVFs in Data Center Lok >2014-04-18 22:36:05,406 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update template OVFs in Data Center Lok >2014-04-18 22:36:05,407 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated templates OVFs in Data Center Lok >2014-04-18 22:36:05,407 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 22:36:05,407 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 22:41:38,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1329fd3b >2014-04-18 22:41:38,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 29b03651 >2014-04-18 22:41:39,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 29b03651 >2014-04-18 22:41:39,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1329fd3b >2014-04-18 22:41:39,036 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-18 23:36:04,501 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-04-18 23:36:05,407 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-04-18 23:36:05,408 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VM OVFs in Data Center Lok >2014-04-18 23:36:05,408 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully updated VM OVFs in Data Center Lok >2014-04-18 23:36:05,408 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update template OVFs in Data Center Lok >2014-04-18 23:36:05,409 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully updated templates OVFs in Data Center Lok >2014-04-18 23:36:05,409 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-18 23:36:05,409 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-18 23:41:39,040 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 371c8e54 >2014-04-18 23:41:39,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 33c44910 >2014-04-18 23:41:39,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 33c44910 >2014-04-18 23:41:39,270 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 371c8e54 >2014-04-18 23:41:39,275 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 00:36:04,503 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-04-19 00:36:05,410 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Attempting to update VMs/Templates Ovf. >2014-04-19 00:36:05,411 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Attempting to update VM OVFs in Data Center Lok >2014-04-19 00:36:05,412 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Successfully updated VM OVFs in Data Center Lok >2014-04-19 00:36:05,412 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Attempting to update template OVFs in Data Center Lok >2014-04-19 00:36:05,413 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Successfully updated templates OVFs in Data Center Lok >2014-04-19 00:36:05,413 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 00:36:05,414 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 00:41:39,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6b17c71b >2014-04-19 00:41:39,280 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6610f48a >2014-04-19 00:41:39,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6610f48a >2014-04-19 00:41:39,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6b17c71b >2014-04-19 00:41:39,506 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 01:36:04,505 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-04-19 01:36:05,414 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-19 01:36:05,415 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-19 01:36:05,416 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-19 01:36:05,416 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-19 01:36:05,416 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-19 01:36:05,416 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 01:36:05,417 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 01:41:39,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 507244ba >2014-04-19 01:41:39,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7dfe3898 >2014-04-19 01:41:39,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7dfe3898 >2014-04-19 01:41:39,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 507244ba >2014-04-19 01:41:39,745 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 02:36:04,508 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-04-19 02:36:05,418 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VMs/Templates Ovf. >2014-04-19 02:36:05,419 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update VM OVFs in Data Center Lok >2014-04-19 02:36:05,419 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated VM OVFs in Data Center Lok >2014-04-19 02:36:05,419 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to update template OVFs in Data Center Lok >2014-04-19 02:36:05,420 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully updated templates OVFs in Data Center Lok >2014-04-19 02:36:05,420 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 02:36:05,421 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-74) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 02:41:39,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5acf37a7 >2014-04-19 02:41:39,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5a4ee384 >2014-04-19 02:41:39,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a4ee384 >2014-04-19 02:41:39,976 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5acf37a7 >2014-04-19 02:41:39,980 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-44) Start deleteAgedOutAuditLogs >2014-04-19 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-44) Finished deleteAgedOutAuditLogs >2014-04-19 03:36:04,509 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-04-19 03:36:05,421 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VMs/Templates Ovf. >2014-04-19 03:36:05,422 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update VM OVFs in Data Center Lok >2014-04-19 03:36:05,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully updated VM OVFs in Data Center Lok >2014-04-19 03:36:05,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to update template OVFs in Data Center Lok >2014-04-19 03:36:05,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully updated templates OVFs in Data Center Lok >2014-04-19 03:36:05,423 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 03:36:05,424 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-83) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 03:41:39,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) [6d63ed15] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 65ad0acb >2014-04-19 03:41:39,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) [6d63ed15] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 439fc993 >2014-04-19 03:41:40,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) [6d63ed15] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 439fc993 >2014-04-19 03:41:40,221 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) [6d63ed15] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 65ad0acb >2014-04-19 03:41:40,226 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) [6d63ed15] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 04:36:04,511 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-18) Start refreshing all users data >2014-04-19 04:36:05,424 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VMs/Templates Ovf. >2014-04-19 04:36:05,425 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update VM OVFs in Data Center Lok >2014-04-19 04:36:05,426 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated VM OVFs in Data Center Lok >2014-04-19 04:36:05,426 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to update template OVFs in Data Center Lok >2014-04-19 04:36:05,428 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully updated templates OVFs in Data Center Lok >2014-04-19 04:36:05,428 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 04:36:05,432 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 04:41:40,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 25fd09c >2014-04-19 04:41:40,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6078b2fb >2014-04-19 04:41:40,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6078b2fb >2014-04-19 04:41:40,455 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 25fd09c >2014-04-19 04:41:40,461 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 05:36:04,512 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-19 05:36:05,432 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-04-19 05:36:05,433 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VM OVFs in Data Center Lok >2014-04-19 05:36:05,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated VM OVFs in Data Center Lok >2014-04-19 05:36:05,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update template OVFs in Data Center Lok >2014-04-19 05:36:05,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated templates OVFs in Data Center Lok >2014-04-19 05:36:05,434 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 05:36:05,435 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 05:41:40,465 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 17553119 >2014-04-19 05:41:40,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5a0a6c46 >2014-04-19 05:41:40,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a0a6c46 >2014-04-19 05:41:40,704 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 17553119 >2014-04-19 05:41:40,710 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 06:36:04,514 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-04-19 06:36:05,435 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-04-19 06:36:05,436 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VM OVFs in Data Center Lok >2014-04-19 06:36:05,437 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated VM OVFs in Data Center Lok >2014-04-19 06:36:05,437 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update template OVFs in Data Center Lok >2014-04-19 06:36:05,438 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated templates OVFs in Data Center Lok >2014-04-19 06:36:05,438 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 06:36:05,438 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 06:41:40,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 9c86ae >2014-04-19 06:41:40,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6213d147 >2014-04-19 06:41:41,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6213d147 >2014-04-19 06:41:41,199 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 9c86ae >2014-04-19 06:41:41,204 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 07:36:04,516 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-04-19 07:36:05,438 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-04-19 07:36:05,439 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VM OVFs in Data Center Lok >2014-04-19 07:36:05,440 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated VM OVFs in Data Center Lok >2014-04-19 07:36:05,440 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update template OVFs in Data Center Lok >2014-04-19 07:36:05,440 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated templates OVFs in Data Center Lok >2014-04-19 07:36:05,440 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 07:36:05,441 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 07:41:41,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 87716d4 >2014-04-19 07:41:41,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 427bbca3 >2014-04-19 07:41:41,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 427bbca3 >2014-04-19 07:41:41,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 87716d4 >2014-04-19 07:41:41,436 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 08:36:04,517 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-04-19 08:36:05,441 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-04-19 08:36:05,442 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VM OVFs in Data Center Lok >2014-04-19 08:36:05,443 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated VM OVFs in Data Center Lok >2014-04-19 08:36:05,443 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update template OVFs in Data Center Lok >2014-04-19 08:36:05,444 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated templates OVFs in Data Center Lok >2014-04-19 08:36:05,444 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 08:36:05,444 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 08:41:41,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1122301b >2014-04-19 08:41:41,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5b844566 >2014-04-19 08:41:41,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5b844566 >2014-04-19 08:41:41,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1122301b >2014-04-19 08:41:41,672 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 09:36:04,519 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-04-19 09:36:05,444 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-19 09:36:05,445 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-19 09:36:05,446 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-19 09:36:05,446 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-19 09:36:05,446 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-19 09:36:05,446 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 09:36:05,447 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 09:41:41,676 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6ed2c8bb >2014-04-19 09:41:41,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5078a5d4 >2014-04-19 09:41:41,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5078a5d4 >2014-04-19 09:41:41,898 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ed2c8bb >2014-04-19 09:41:41,903 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 10:36:04,520 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-04-19 10:36:05,447 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-04-19 10:36:05,448 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VM OVFs in Data Center Lok >2014-04-19 10:36:05,449 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated VM OVFs in Data Center Lok >2014-04-19 10:36:05,449 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update template OVFs in Data Center Lok >2014-04-19 10:36:05,450 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated templates OVFs in Data Center Lok >2014-04-19 10:36:05,450 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 10:36:05,450 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 10:41:41,906 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 770758f8 >2014-04-19 10:41:41,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3623ed5 >2014-04-19 10:41:42,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3623ed5 >2014-04-19 10:41:42,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 770758f8 >2014-04-19 10:41:42,145 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 11:36:04,522 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-04-19 11:36:05,450 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-04-19 11:36:05,451 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VM OVFs in Data Center Lok >2014-04-19 11:36:05,452 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated VM OVFs in Data Center Lok >2014-04-19 11:36:05,452 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update template OVFs in Data Center Lok >2014-04-19 11:36:05,452 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated templates OVFs in Data Center Lok >2014-04-19 11:36:05,452 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 11:36:05,453 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 11:41:42,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 11560894 >2014-04-19 11:41:42,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 274d3570 >2014-04-19 11:41:42,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 274d3570 >2014-04-19 11:41:42,386 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 11560894 >2014-04-19 11:41:42,391 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 12:36:04,524 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-04-19 12:36:05,453 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-04-19 12:36:05,454 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VM OVFs in Data Center Lok >2014-04-19 12:36:05,455 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated VM OVFs in Data Center Lok >2014-04-19 12:36:05,455 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update template OVFs in Data Center Lok >2014-04-19 12:36:05,455 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated templates OVFs in Data Center Lok >2014-04-19 12:36:05,455 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 12:36:05,456 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 12:41:42,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: a30b46c >2014-04-19 12:41:42,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 39523ae >2014-04-19 12:41:42,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 39523ae >2014-04-19 12:41:42,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: a30b46c >2014-04-19 12:41:42,623 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 13:36:04,526 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-04-19 13:36:05,456 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-04-19 13:36:05,457 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VM OVFs in Data Center Lok >2014-04-19 13:36:05,458 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated VM OVFs in Data Center Lok >2014-04-19 13:36:05,458 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update template OVFs in Data Center Lok >2014-04-19 13:36:05,458 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated templates OVFs in Data Center Lok >2014-04-19 13:36:05,459 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 13:36:05,459 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 13:41:42,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 50896b35 >2014-04-19 13:41:42,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6ddd8de1 >2014-04-19 13:41:42,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ddd8de1 >2014-04-19 13:41:42,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 50896b35 >2014-04-19 13:41:42,856 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 14:36:04,528 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-04-19 14:36:05,460 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-04-19 14:36:05,461 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VM OVFs in Data Center Lok >2014-04-19 14:36:05,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated VM OVFs in Data Center Lok >2014-04-19 14:36:05,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update template OVFs in Data Center Lok >2014-04-19 14:36:05,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated templates OVFs in Data Center Lok >2014-04-19 14:36:05,462 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 14:36:05,463 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 14:41:42,863 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 21e090f6 >2014-04-19 14:41:42,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1e4d17da >2014-04-19 14:41:43,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1e4d17da >2014-04-19 14:41:43,087 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 21e090f6 >2014-04-19 14:41:43,092 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 15:36:04,529 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-04-19 15:36:05,463 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update VMs/Templates Ovf. >2014-04-19 15:36:05,464 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update VM OVFs in Data Center Lok >2014-04-19 15:36:05,465 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully updated VM OVFs in Data Center Lok >2014-04-19 15:36:05,465 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update template OVFs in Data Center Lok >2014-04-19 15:36:05,466 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully updated templates OVFs in Data Center Lok >2014-04-19 15:36:05,466 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 15:36:05,466 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 15:41:43,100 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 571e189b >2014-04-19 15:41:43,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 78e194cb >2014-04-19 15:41:43,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 78e194cb >2014-04-19 15:41:43,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 571e189b >2014-04-19 15:41:43,496 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 16:36:04,531 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-04-19 16:36:05,467 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-04-19 16:36:05,468 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-04-19 16:36:05,468 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-04-19 16:36:05,468 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-04-19 16:36:05,469 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-04-19 16:36:05,469 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 16:36:05,469 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 16:41:43,502 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5b61e3bb >2014-04-19 16:41:43,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 77ae38db >2014-04-19 16:41:43,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 77ae38db >2014-04-19 16:41:43,734 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5b61e3bb >2014-04-19 16:41:43,744 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 17:36:04,533 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-04-19 17:36:05,470 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-04-19 17:36:05,472 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VM OVFs in Data Center Lok >2014-04-19 17:36:05,473 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated VM OVFs in Data Center Lok >2014-04-19 17:36:05,473 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update template OVFs in Data Center Lok >2014-04-19 17:36:05,474 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated templates OVFs in Data Center Lok >2014-04-19 17:36:05,474 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 17:36:05,474 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 17:41:43,753 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 16d46ba6 >2014-04-19 17:41:43,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3a4dd1e2 >2014-04-19 17:41:44,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3a4dd1e2 >2014-04-19 17:41:44,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 16d46ba6 >2014-04-19 17:41:44,201 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 18:36:04,535 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-19 18:36:05,474 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-04-19 18:36:05,475 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VM OVFs in Data Center Lok >2014-04-19 18:36:05,476 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated VM OVFs in Data Center Lok >2014-04-19 18:36:05,476 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update template OVFs in Data Center Lok >2014-04-19 18:36:05,476 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated templates OVFs in Data Center Lok >2014-04-19 18:36:05,476 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 18:36:05,477 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 18:41:44,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: e1a2cd1 >2014-04-19 18:41:44,209 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 58745492 >2014-04-19 18:41:44,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 58745492 >2014-04-19 18:41:44,445 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: e1a2cd1 >2014-04-19 18:41:44,450 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 19:36:04,536 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-57) Start refreshing all users data >2014-04-19 19:36:05,478 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-04-19 19:36:05,478 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VM OVFs in Data Center Lok >2014-04-19 19:36:05,479 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated VM OVFs in Data Center Lok >2014-04-19 19:36:05,479 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update template OVFs in Data Center Lok >2014-04-19 19:36:05,480 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated templates OVFs in Data Center Lok >2014-04-19 19:36:05,480 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 19:36:05,480 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 19:41:44,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: f39ce1e >2014-04-19 19:41:44,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1c0b1b61 >2014-04-19 19:41:44,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1c0b1b61 >2014-04-19 19:41:44,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: f39ce1e >2014-04-19 19:41:44,705 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 20:36:04,537 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-04-19 20:36:05,480 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-04-19 20:36:05,481 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VM OVFs in Data Center Lok >2014-04-19 20:36:05,482 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated VM OVFs in Data Center Lok >2014-04-19 20:36:05,482 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update template OVFs in Data Center Lok >2014-04-19 20:36:05,482 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated templates OVFs in Data Center Lok >2014-04-19 20:36:05,482 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 20:36:05,483 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 20:41:44,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 20defa72 >2014-04-19 20:41:44,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: f0c633a >2014-04-19 20:41:44,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: f0c633a >2014-04-19 20:41:44,950 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 20defa72 >2014-04-19 20:41:44,957 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 21:36:04,539 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-04-19 21:36:05,483 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-19 21:36:05,484 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center Lok >2014-04-19 21:36:05,485 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center Lok >2014-04-19 21:36:05,485 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center Lok >2014-04-19 21:36:05,486 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center Lok >2014-04-19 21:36:05,486 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 21:36:05,486 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 21:41:44,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5bc4ed9e >2014-04-19 21:41:44,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5dcc4f23 >2014-04-19 21:41:45,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5dcc4f23 >2014-04-19 21:41:45,196 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5bc4ed9e >2014-04-19 21:41:45,203 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 22:36:04,541 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Start refreshing all users data >2014-04-19 22:36:05,486 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-04-19 22:36:05,488 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VM OVFs in Data Center Lok >2014-04-19 22:36:05,489 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated VM OVFs in Data Center Lok >2014-04-19 22:36:05,489 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update template OVFs in Data Center Lok >2014-04-19 22:36:05,489 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated templates OVFs in Data Center Lok >2014-04-19 22:36:05,489 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 22:36:05,490 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 22:41:45,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 710da83f >2014-04-19 22:41:45,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 37a0fa15 >2014-04-19 22:41:45,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 37a0fa15 >2014-04-19 22:41:45,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 710da83f >2014-04-19 22:41:45,447 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-19 23:36:04,542 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-04-19 23:36:05,490 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-19 23:36:05,491 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-19 23:36:05,492 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-19 23:36:05,492 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-19 23:36:05,493 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-19 23:36:05,493 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-19 23:36:05,496 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-19 23:41:45,452 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1b3c71af >2014-04-19 23:41:45,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1d4ba621 >2014-04-19 23:41:45,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d4ba621 >2014-04-19 23:41:45,682 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b3c71af >2014-04-19 23:41:45,688 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 00:36:04,543 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-46) Start refreshing all users data >2014-04-20 00:36:05,496 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-20 00:36:05,498 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-20 00:36:05,499 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-20 00:36:05,499 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-20 00:36:05,500 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-20 00:36:05,500 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 00:36:05,500 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 00:41:45,693 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1deeda10 >2014-04-20 00:41:45,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2c2e5fca >2014-04-20 00:41:45,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2c2e5fca >2014-04-20 00:41:45,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1deeda10 >2014-04-20 00:41:45,934 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 01:36:04,545 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-6) Start refreshing all users data >2014-04-20 01:36:05,500 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-04-20 01:36:05,500 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VM OVFs in Data Center Lok >2014-04-20 01:36:05,501 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated VM OVFs in Data Center Lok >2014-04-20 01:36:05,501 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update template OVFs in Data Center Lok >2014-04-20 01:36:05,502 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated templates OVFs in Data Center Lok >2014-04-20 01:36:05,502 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 01:36:05,502 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 01:41:45,938 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4c2c2136 >2014-04-20 01:41:45,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6538874b >2014-04-20 01:41:46,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6538874b >2014-04-20 01:41:46,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4c2c2136 >2014-04-20 01:41:46,181 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 02:36:04,547 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-20 02:36:05,503 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-04-20 02:36:05,504 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VM OVFs in Data Center Lok >2014-04-20 02:36:05,504 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated VM OVFs in Data Center Lok >2014-04-20 02:36:05,504 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update template OVFs in Data Center Lok >2014-04-20 02:36:05,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated templates OVFs in Data Center Lok >2014-04-20 02:36:05,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 02:36:05,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 02:41:46,185 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7706afc9 >2014-04-20 02:41:46,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 57fef22b >2014-04-20 02:41:46,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 57fef22b >2014-04-20 02:41:46,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7706afc9 >2014-04-20 02:41:46,429 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-10) Start deleteAgedOutAuditLogs >2014-04-20 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-10) Finished deleteAgedOutAuditLogs >2014-04-20 03:36:04,549 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-04-20 03:36:05,505 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-04-20 03:36:05,506 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Lok >2014-04-20 03:36:05,507 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Lok >2014-04-20 03:36:05,507 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Lok >2014-04-20 03:36:05,507 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated templates OVFs in Data Center Lok >2014-04-20 03:36:05,507 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 03:36:05,508 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 03:41:46,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: ea7d2a7 >2014-04-20 03:41:46,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2bf6d975 >2014-04-20 03:41:46,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2bf6d975 >2014-04-20 03:41:46,666 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ea7d2a7 >2014-04-20 03:41:46,671 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 04:36:04,550 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-20 04:36:05,509 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-04-20 04:36:05,510 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VM OVFs in Data Center Lok >2014-04-20 04:36:05,510 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated VM OVFs in Data Center Lok >2014-04-20 04:36:05,511 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update template OVFs in Data Center Lok >2014-04-20 04:36:05,511 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated templates OVFs in Data Center Lok >2014-04-20 04:36:05,511 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 04:36:05,512 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 04:41:46,675 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 273a6907 >2014-04-20 04:41:46,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6bc55eaf >2014-04-20 04:41:46,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6bc55eaf >2014-04-20 04:41:46,913 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 273a6907 >2014-04-20 04:41:46,917 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 05:36:04,552 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-57) Start refreshing all users data >2014-04-20 05:36:05,512 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-20 05:36:05,513 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-20 05:36:05,514 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-20 05:36:05,514 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-20 05:36:05,514 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-20 05:36:05,514 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 05:36:05,515 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 05:41:46,920 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6b44ae1d >2014-04-20 05:41:46,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6a0328ef >2014-04-20 05:41:47,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6a0328ef >2014-04-20 05:41:47,152 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6b44ae1d >2014-04-20 05:41:47,156 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 06:36:04,554 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-04-20 06:36:05,516 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VMs/Templates Ovf. >2014-04-20 06:36:05,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update VM OVFs in Data Center Lok >2014-04-20 06:36:05,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Successfully updated VM OVFs in Data Center Lok >2014-04-20 06:36:05,517 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to update template OVFs in Data Center Lok >2014-04-20 06:36:05,518 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Successfully updated templates OVFs in Data Center Lok >2014-04-20 06:36:05,518 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 06:36:05,518 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 06:41:47,159 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 26625092 >2014-04-20 06:41:47,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 97b4ee >2014-04-20 06:41:47,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 97b4ee >2014-04-20 06:41:47,388 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 26625092 >2014-04-20 06:41:47,393 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 07:36:04,556 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-04-20 07:36:05,519 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-04-20 07:36:05,520 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VM OVFs in Data Center Lok >2014-04-20 07:36:05,521 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully updated VM OVFs in Data Center Lok >2014-04-20 07:36:05,521 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update template OVFs in Data Center Lok >2014-04-20 07:36:05,526 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully updated templates OVFs in Data Center Lok >2014-04-20 07:36:05,526 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 07:36:05,527 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 07:41:47,397 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 46506c37 >2014-04-20 07:41:47,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 49f0b2f1 >2014-04-20 07:41:47,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 49f0b2f1 >2014-04-20 07:41:47,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 46506c37 >2014-04-20 07:41:47,623 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 08:36:04,557 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-04-20 08:36:05,528 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VMs/Templates Ovf. >2014-04-20 08:36:05,529 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update VM OVFs in Data Center Lok >2014-04-20 08:36:05,529 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully updated VM OVFs in Data Center Lok >2014-04-20 08:36:05,529 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to update template OVFs in Data Center Lok >2014-04-20 08:36:05,530 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully updated templates OVFs in Data Center Lok >2014-04-20 08:36:05,530 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 08:36:05,530 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-56) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 08:41:47,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 70c1ebbc >2014-04-20 08:41:47,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4b5bbd79 >2014-04-20 08:41:47,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b5bbd79 >2014-04-20 08:41:47,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 70c1ebbc >2014-04-20 08:41:47,856 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 09:36:04,559 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-04-20 09:36:05,531 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-04-20 09:36:05,532 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-04-20 09:36:05,532 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-04-20 09:36:05,532 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-04-20 09:36:05,533 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-04-20 09:36:05,533 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 09:36:05,537 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 09:41:47,860 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4abf8004 >2014-04-20 09:41:47,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 626dacf2 >2014-04-20 09:41:48,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 626dacf2 >2014-04-20 09:41:48,085 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4abf8004 >2014-04-20 09:41:48,093 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 10:36:04,561 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-04-20 10:36:05,537 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-20 10:36:05,539 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-04-20 10:36:05,539 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-04-20 10:36:05,539 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-04-20 10:36:05,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-04-20 10:36:05,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 10:36:05,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 10:41:48,097 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6bcb42b2 >2014-04-20 10:41:48,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6d87eb67 >2014-04-20 10:41:48,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6d87eb67 >2014-04-20 10:41:48,491 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6bcb42b2 >2014-04-20 10:41:48,496 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 11:36:04,563 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-04-20 11:36:05,540 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-04-20 11:36:05,541 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-04-20 11:36:05,542 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-04-20 11:36:05,542 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-04-20 11:36:05,542 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-04-20 11:36:05,542 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 11:36:05,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 11:41:48,500 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4c7ca64e >2014-04-20 11:41:48,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: b8a721d >2014-04-20 11:41:48,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: b8a721d >2014-04-20 11:41:48,729 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4c7ca64e >2014-04-20 11:41:48,738 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 12:36:04,565 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-20 12:36:05,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VMs/Templates Ovf. >2014-04-20 12:36:05,544 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update VM OVFs in Data Center Lok >2014-04-20 12:36:05,545 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated VM OVFs in Data Center Lok >2014-04-20 12:36:05,545 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to update template OVFs in Data Center Lok >2014-04-20 12:36:05,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully updated templates OVFs in Data Center Lok >2014-04-20 12:36:05,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 12:36:05,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-75) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 12:41:48,741 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4ee1ee89 >2014-04-20 12:41:48,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 52f38cd4 >2014-04-20 12:41:48,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 52f38cd4 >2014-04-20 12:41:48,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4ee1ee89 >2014-04-20 12:41:48,969 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 13:36:04,567 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-04-20 13:36:05,546 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VMs/Templates Ovf. >2014-04-20 13:36:05,547 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update VM OVFs in Data Center Lok >2014-04-20 13:36:05,548 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated VM OVFs in Data Center Lok >2014-04-20 13:36:05,548 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to update template OVFs in Data Center Lok >2014-04-20 13:36:05,548 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully updated templates OVFs in Data Center Lok >2014-04-20 13:36:05,548 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 13:36:05,549 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-10) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 13:41:48,973 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 201efb40 >2014-04-20 13:41:48,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 52013739 >2014-04-20 13:41:49,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 52013739 >2014-04-20 13:41:49,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 201efb40 >2014-04-20 13:41:49,206 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 14:36:04,569 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-04-20 14:36:05,549 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-04-20 14:36:05,550 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VM OVFs in Data Center Lok >2014-04-20 14:36:05,551 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated VM OVFs in Data Center Lok >2014-04-20 14:36:05,551 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update template OVFs in Data Center Lok >2014-04-20 14:36:05,560 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated templates OVFs in Data Center Lok >2014-04-20 14:36:05,560 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 14:36:05,561 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 14:41:49,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1c2f6696 >2014-04-20 14:41:49,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5aac5915 >2014-04-20 14:41:49,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5aac5915 >2014-04-20 14:41:49,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1c2f6696 >2014-04-20 14:41:49,566 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 15:36:04,571 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-04-20 15:36:05,561 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-20 15:36:05,562 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-20 15:36:05,562 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-20 15:36:05,563 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-20 15:36:05,563 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-20 15:36:05,563 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 15:36:05,564 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 15:41:49,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7855b1e >2014-04-20 15:41:49,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4421126d >2014-04-20 15:41:49,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4421126d >2014-04-20 15:41:49,799 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7855b1e >2014-04-20 15:41:49,804 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 16:36:04,573 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-20) Start refreshing all users data >2014-04-20 16:36:05,565 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-04-20 16:36:05,565 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VM OVFs in Data Center Lok >2014-04-20 16:36:05,566 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated VM OVFs in Data Center Lok >2014-04-20 16:36:05,566 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update template OVFs in Data Center Lok >2014-04-20 16:36:05,567 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated templates OVFs in Data Center Lok >2014-04-20 16:36:05,567 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 16:36:05,567 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 16:41:49,807 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4b681a4b >2014-04-20 16:41:49,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 315dfe9 >2014-04-20 16:41:50,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 315dfe9 >2014-04-20 16:41:50,181 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b681a4b >2014-04-20 16:41:50,187 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 17:36:04,575 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-20 17:36:05,567 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-04-20 17:36:05,568 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VM OVFs in Data Center Lok >2014-04-20 17:36:05,569 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated VM OVFs in Data Center Lok >2014-04-20 17:36:05,569 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update template OVFs in Data Center Lok >2014-04-20 17:36:05,569 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated templates OVFs in Data Center Lok >2014-04-20 17:36:05,569 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 17:36:05,570 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 17:41:50,191 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4b9f8b1f >2014-04-20 17:41:50,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5766c702 >2014-04-20 17:41:50,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5766c702 >2014-04-20 17:41:50,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b9f8b1f >2014-04-20 17:41:50,431 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 18:36:04,577 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-04-20 18:36:05,570 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-04-20 18:36:05,571 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VM OVFs in Data Center Lok >2014-04-20 18:36:05,572 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated VM OVFs in Data Center Lok >2014-04-20 18:36:05,572 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update template OVFs in Data Center Lok >2014-04-20 18:36:05,572 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated templates OVFs in Data Center Lok >2014-04-20 18:36:05,572 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 18:36:05,573 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 18:41:50,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 47256a41 >2014-04-20 18:41:50,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6962f5e4 >2014-04-20 18:41:50,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6962f5e4 >2014-04-20 18:41:50,661 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 47256a41 >2014-04-20 18:41:50,665 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 19:36:04,579 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-20 19:36:05,574 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-20 19:36:05,575 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-20 19:36:05,576 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-20 19:36:05,576 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-20 19:36:05,576 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-20 19:36:05,576 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 19:36:05,577 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 19:41:50,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5981a005 >2014-04-20 19:41:50,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4d30d62b >2014-04-20 19:41:50,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4d30d62b >2014-04-20 19:41:50,896 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5981a005 >2014-04-20 19:41:50,900 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 20:36:04,580 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-04-20 20:36:05,577 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-04-20 20:36:05,578 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Lok >2014-04-20 20:36:05,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Lok >2014-04-20 20:36:05,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Lok >2014-04-20 20:36:05,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated templates OVFs in Data Center Lok >2014-04-20 20:36:05,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 20:36:05,580 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 20:41:50,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 465ccd41 >2014-04-20 20:41:50,905 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 76a78a7d >2014-04-20 20:41:51,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76a78a7d >2014-04-20 20:41:51,128 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 465ccd41 >2014-04-20 20:41:51,134 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 21:36:04,581 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-41) Start refreshing all users data >2014-04-20 21:36:05,580 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-20 21:36:05,581 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-20 21:36:05,582 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-20 21:36:05,582 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-20 21:36:05,582 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-20 21:36:05,582 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 21:36:05,583 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 21:41:51,138 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 16070e42 >2014-04-20 21:41:51,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 29391a5a >2014-04-20 21:41:51,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 29391a5a >2014-04-20 21:41:51,363 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 16070e42 >2014-04-20 21:41:51,370 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 22:36:04,583 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-04-20 22:36:05,583 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-04-20 22:36:05,584 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VM OVFs in Data Center Lok >2014-04-20 22:36:05,584 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated VM OVFs in Data Center Lok >2014-04-20 22:36:05,584 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update template OVFs in Data Center Lok >2014-04-20 22:36:05,585 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated templates OVFs in Data Center Lok >2014-04-20 22:36:05,585 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 22:36:05,586 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 22:41:51,374 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 29c049c8 >2014-04-20 22:41:51,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: f2bfee2 >2014-04-20 22:41:51,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: f2bfee2 >2014-04-20 22:41:51,600 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 29c049c8 >2014-04-20 22:41:51,605 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-20 23:36:04,585 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-04-20 23:36:05,586 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VMs/Templates Ovf. >2014-04-20 23:36:05,587 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update VM OVFs in Data Center Lok >2014-04-20 23:36:05,587 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated VM OVFs in Data Center Lok >2014-04-20 23:36:05,588 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to update template OVFs in Data Center Lok >2014-04-20 23:36:05,588 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully updated templates OVFs in Data Center Lok >2014-04-20 23:36:05,588 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-20 23:36:05,589 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-69) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-20 23:41:51,609 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2a0299fd >2014-04-20 23:41:51,609 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 10cbc697 >2014-04-20 23:41:51,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 10cbc697 >2014-04-20 23:41:51,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2a0299fd >2014-04-20 23:41:51,840 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 00:36:04,587 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-04-21 00:36:05,589 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-04-21 00:36:05,590 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VM OVFs in Data Center Lok >2014-04-21 00:36:05,590 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated VM OVFs in Data Center Lok >2014-04-21 00:36:05,591 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update template OVFs in Data Center Lok >2014-04-21 00:36:05,591 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated templates OVFs in Data Center Lok >2014-04-21 00:36:05,591 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 00:36:05,592 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 00:41:51,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 23314dd8 >2014-04-21 00:41:51,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 79edb1f4 >2014-04-21 00:41:52,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 79edb1f4 >2014-04-21 00:41:52,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23314dd8 >2014-04-21 00:41:52,075 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 01:36:04,589 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-04-21 01:36:05,592 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-04-21 01:36:05,593 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VM OVFs in Data Center Lok >2014-04-21 01:36:05,594 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated VM OVFs in Data Center Lok >2014-04-21 01:36:05,594 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update template OVFs in Data Center Lok >2014-04-21 01:36:05,595 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated templates OVFs in Data Center Lok >2014-04-21 01:36:05,595 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 01:36:05,596 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 01:41:52,080 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3ffa5ab0 >2014-04-21 01:41:52,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4d57f9a0 >2014-04-21 01:41:52,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4d57f9a0 >2014-04-21 01:41:52,318 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3ffa5ab0 >2014-04-21 01:41:52,325 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 02:36:04,591 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-04-21 02:36:05,596 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-04-21 02:36:05,597 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VM OVFs in Data Center Lok >2014-04-21 02:36:05,598 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated VM OVFs in Data Center Lok >2014-04-21 02:36:05,598 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update template OVFs in Data Center Lok >2014-04-21 02:36:05,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated templates OVFs in Data Center Lok >2014-04-21 02:36:05,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 02:36:05,600 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 02:41:52,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d30825b >2014-04-21 02:41:52,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5bbe3355 >2014-04-21 02:41:52,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5bbe3355 >2014-04-21 02:41:52,552 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d30825b >2014-04-21 02:41:52,557 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-29) Start deleteAgedOutAuditLogs >2014-04-21 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-29) Finished deleteAgedOutAuditLogs >2014-04-21 03:36:04,593 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-04-21 03:36:05,600 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-04-21 03:36:05,601 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VM OVFs in Data Center Lok >2014-04-21 03:36:05,602 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated VM OVFs in Data Center Lok >2014-04-21 03:36:05,602 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update template OVFs in Data Center Lok >2014-04-21 03:36:05,602 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated templates OVFs in Data Center Lok >2014-04-21 03:36:05,602 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 03:36:05,603 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 03:41:52,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4beae641 >2014-04-21 03:41:52,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2ab2dc57 >2014-04-21 03:41:52,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2ab2dc57 >2014-04-21 03:41:52,785 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4beae641 >2014-04-21 03:41:52,790 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 03:55:29,098 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ListVDSCommand] (DefaultQuartzScheduler_Worker-51) Command ListVDS execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-21 03:55:29,108 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-51) Failed to refresh VDS , vds = 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-04-21 03:55:34,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-87) Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-21 03:55:34,115 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-87) Failed to refresh VDS , vds = 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209, VDS Network Error, continuing. >java.net.SocketTimeoutException: connect timed out >2014-04-21 03:55:37,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 743f7b51 >2014-04-21 03:55:39,401 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) Command GetHardwareInfoVDS execution failed. Exception: VDSNetworkException: java.net.SocketTimeoutException: connect timed out >2014-04-21 03:55:39,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) FINISH, GetHardwareInfoVDSCommand, log id: 743f7b51 >2014-04-21 03:55:39,407 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.net.SocketTimeoutException: connect timed out > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:102) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >Caused by: java.net.SocketTimeoutException: connect timed out > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:618) > at org.ovirt.engine.core.utils.ssl.AuthSSLProtocolSocketFactory.createSocket(AuthSSLProtocolSocketFactory.java:288) > at org.apache.commons.httpclient.HttpConnection.open(HttpConnection.java:707) > at org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$HttpConnectionAdapter.open(MultiThreadedHttpConnectionManager.java:1361) > at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:387) > at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397) > at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323) > at org.apache.xmlrpc.client.XmlRpcCommonsTransport.writeRequest(XmlRpcCommonsTransport.java:227) > at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:151) > at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) > at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) > at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) > at org.apache.xmlrpc.client.util.ClientFactory$1.invoke(ClientFactory.java:140) > at com.sun.proxy.$Proxy59.getVdsHardwareInfo(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.vdsbroker.xmlrpc.XmlRpcUtils$AsyncProxy$InternalCallable.call(XmlRpcUtils.java:225) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-21 03:55:39,422 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-21 03:55:39,638 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-33) [34512b44] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-21 03:55:39,642 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-33) [6135fe8a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-21 03:55:39,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-33) [6135fe8a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, storageType = LOCALFS, connectionList = [{ id: 48bb8365-f600-4eba-8ed7-e823810d8790, connection: /mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6cec114e >2014-04-21 03:55:46,905 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) Command SpmStatusVDS execution failed. Exception: VDSNetworkException: java.net.NoRouteToHostException: No route to host >2014-04-21 03:55:46,907 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-21 03:55:46,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Correlation ID: 35b64ebe, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Lok. Setting Data Center status to Non Responsive (On host 10.34.27.209, Error: Network error during communication with the Host.). >2014-04-21 03:55:46,921 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-88) [35b64ebe] IrsBroker::Failed::GetStoragePoolInfoVDS due to: NoRouteToHostException: No route to host >2014-04-21 03:55:55,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-33) [6135fe8a] FINISH, ConnectStorageServerVDSCommand, return: {48bb8365-f600-4eba-8ed7-e823810d8790=0}, log id: 6cec114e >2014-04-21 03:55:55,096 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-33) [6135fe8a] Host 10.34.27.209 storage connection was succeeded >2014-04-21 03:55:55,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-33) [6135fe8a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1bac2511 >2014-04-21 03:55:55,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-33) [6135fe8a] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 1bac2511 >2014-04-21 03:55:55,318 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-33) [6135fe8a] Host 10.34.27.209 storage connection was succeeded >2014-04-21 03:55:55,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-33) [6135fe8a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7670f9af >2014-04-21 03:55:55,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-33) [6135fe8a] FINISH, ConnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 7670f9af >2014-04-21 03:55:55,541 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-33) [6135fe8a] Host 10.34.27.209 storage connection was succeeded >2014-04-21 03:55:55,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, vds_spm_id = 1, masterDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterVersion = 1), log id: 7383ef28 >2014-04-21 03:56:03,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7383ef28 >2014-04-21 03:56:03,342 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [6135fe8a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-21 03:56:03,402 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-33) [670e2dc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-21 03:56:03,410 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-33) [431634a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-21 04:36:04,594 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-04-21 04:36:05,604 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-04-21 04:36:05,604 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-04-21 04:36:05,605 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-04-21 04:36:05,605 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-04-21 04:36:05,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-04-21 04:36:05,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 04:36:05,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 04:41:52,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 25af144a >2014-04-21 04:41:52,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: bfa20b7 >2014-04-21 04:41:53,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: bfa20b7 >2014-04-21 04:41:53,201 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 25af144a >2014-04-21 04:41:53,209 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 05:36:04,596 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-6) Start refreshing all users data >2014-04-21 05:36:05,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-21 05:36:05,607 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-21 05:36:05,608 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-21 05:36:05,608 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-21 05:36:05,608 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-21 05:36:05,608 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 05:36:05,609 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 05:41:53,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 405e9d57 >2014-04-21 05:41:53,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3abdbc5 >2014-04-21 05:41:53,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3abdbc5 >2014-04-21 05:41:53,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 405e9d57 >2014-04-21 05:41:53,441 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 06:36:04,598 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Start refreshing all users data >2014-04-21 06:36:05,609 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-04-21 06:36:05,610 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VM OVFs in Data Center Lok >2014-04-21 06:36:05,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated VM OVFs in Data Center Lok >2014-04-21 06:36:05,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update template OVFs in Data Center Lok >2014-04-21 06:36:05,612 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated templates OVFs in Data Center Lok >2014-04-21 06:36:05,612 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 06:36:05,613 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 06:41:53,444 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 59401817 >2014-04-21 06:41:53,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7eedbbc7 >2014-04-21 06:41:53,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7eedbbc7 >2014-04-21 06:41:53,668 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 59401817 >2014-04-21 06:41:53,674 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 07:36:04,599 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-21 07:36:05,613 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-21 07:36:05,614 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-21 07:36:05,615 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-21 07:36:05,615 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-21 07:36:05,615 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-21 07:36:05,615 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 07:36:05,616 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 07:41:53,678 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3cf83a42 >2014-04-21 07:41:53,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4cb8e253 >2014-04-21 07:41:53,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4cb8e253 >2014-04-21 07:41:53,901 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3cf83a42 >2014-04-21 07:41:53,906 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 08:36:04,601 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-28) Start refreshing all users data >2014-04-21 08:36:05,617 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-04-21 08:36:05,618 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VM OVFs in Data Center Lok >2014-04-21 08:36:05,619 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully updated VM OVFs in Data Center Lok >2014-04-21 08:36:05,619 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update template OVFs in Data Center Lok >2014-04-21 08:36:05,620 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully updated templates OVFs in Data Center Lok >2014-04-21 08:36:05,620 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 08:36:05,621 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 08:41:53,910 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 69f8d16d >2014-04-21 08:41:53,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4e7d8175 >2014-04-21 08:41:54,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4e7d8175 >2014-04-21 08:41:54,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 69f8d16d >2014-04-21 08:41:54,154 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 09:36:04,603 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-20) Start refreshing all users data >2014-04-21 09:36:05,622 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VMs/Templates Ovf. >2014-04-21 09:36:05,623 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update VM OVFs in Data Center Lok >2014-04-21 09:36:05,623 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated VM OVFs in Data Center Lok >2014-04-21 09:36:05,623 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to update template OVFs in Data Center Lok >2014-04-21 09:36:05,624 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully updated templates OVFs in Data Center Lok >2014-04-21 09:36:05,624 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 09:36:05,624 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-54) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 09:41:54,156 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 51eb5fef >2014-04-21 09:41:54,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5b991541 >2014-04-21 09:41:54,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5b991541 >2014-04-21 09:41:54,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 51eb5fef >2014-04-21 09:41:54,618 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 10:36:04,605 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-04-21 10:36:05,624 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VMs/Templates Ovf. >2014-04-21 10:36:05,625 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update VM OVFs in Data Center Lok >2014-04-21 10:36:05,625 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated VM OVFs in Data Center Lok >2014-04-21 10:36:05,625 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to update template OVFs in Data Center Lok >2014-04-21 10:36:05,626 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully updated templates OVFs in Data Center Lok >2014-04-21 10:36:05,626 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 10:36:05,626 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 10:41:54,622 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: df162e6 >2014-04-21 10:41:54,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6293f130 >2014-04-21 10:41:54,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6293f130 >2014-04-21 10:41:54,851 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: df162e6 >2014-04-21 10:41:54,857 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 11:36:04,607 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-04-21 11:36:05,626 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-04-21 11:36:05,627 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VM OVFs in Data Center Lok >2014-04-21 11:36:05,628 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated VM OVFs in Data Center Lok >2014-04-21 11:36:05,628 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update template OVFs in Data Center Lok >2014-04-21 11:36:05,628 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated templates OVFs in Data Center Lok >2014-04-21 11:36:05,628 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 11:36:05,629 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 11:41:54,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 350603ac >2014-04-21 11:41:54,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1ddb2c66 >2014-04-21 11:41:55,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1ddb2c66 >2014-04-21 11:41:55,090 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 350603ac >2014-04-21 11:41:55,095 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 12:36:04,608 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-04-21 12:36:05,630 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-04-21 12:36:05,631 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Lok >2014-04-21 12:36:05,631 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Lok >2014-04-21 12:36:05,631 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Lok >2014-04-21 12:36:05,632 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated templates OVFs in Data Center Lok >2014-04-21 12:36:05,632 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 12:36:05,632 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 12:41:55,099 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 71d6ec5e >2014-04-21 12:41:55,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 553849b5 >2014-04-21 12:41:55,326 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 553849b5 >2014-04-21 12:41:55,326 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 71d6ec5e >2014-04-21 12:41:55,331 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 13:36:04,610 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-04-21 13:36:05,633 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-04-21 13:36:05,634 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VM OVFs in Data Center Lok >2014-04-21 13:36:05,634 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated VM OVFs in Data Center Lok >2014-04-21 13:36:05,635 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update template OVFs in Data Center Lok >2014-04-21 13:36:05,635 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated templates OVFs in Data Center Lok >2014-04-21 13:36:05,635 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 13:36:05,636 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 13:41:55,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 47623fe5 >2014-04-21 13:41:55,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 350f800 >2014-04-21 13:41:55,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 350f800 >2014-04-21 13:41:55,561 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 47623fe5 >2014-04-21 13:41:55,566 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 14:36:04,612 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-04-21 14:36:05,636 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-04-21 14:36:05,637 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VM OVFs in Data Center Lok >2014-04-21 14:36:05,638 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated VM OVFs in Data Center Lok >2014-04-21 14:36:05,638 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update template OVFs in Data Center Lok >2014-04-21 14:36:05,639 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated templates OVFs in Data Center Lok >2014-04-21 14:36:05,639 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 14:36:05,640 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 14:41:55,568 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 37659d9b >2014-04-21 14:41:55,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 704e4786 >2014-04-21 14:41:55,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 704e4786 >2014-04-21 14:41:55,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 37659d9b >2014-04-21 14:41:55,797 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 15:36:04,613 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-04-21 15:36:05,641 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update VMs/Templates Ovf. >2014-04-21 15:36:05,641 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update VM OVFs in Data Center Lok >2014-04-21 15:36:05,642 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully updated VM OVFs in Data Center Lok >2014-04-21 15:36:05,642 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update template OVFs in Data Center Lok >2014-04-21 15:36:05,643 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully updated templates OVFs in Data Center Lok >2014-04-21 15:36:05,643 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 15:36:05,643 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 15:41:55,800 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 20e102ed >2014-04-21 15:41:55,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 14872678 >2014-04-21 15:41:56,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 14872678 >2014-04-21 15:41:56,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 20e102ed >2014-04-21 15:41:56,036 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 16:36:04,615 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Start refreshing all users data >2014-04-21 16:36:05,643 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-21 16:36:05,644 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-21 16:36:05,645 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-21 16:36:05,645 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-21 16:36:05,645 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-21 16:36:05,645 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 16:36:05,646 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 16:41:56,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2a89b9d7 >2014-04-21 16:41:56,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3807e812 >2014-04-21 16:41:56,270 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3807e812 >2014-04-21 16:41:56,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2a89b9d7 >2014-04-21 16:41:56,275 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 17:36:04,616 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-21 17:36:05,646 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-04-21 17:36:05,647 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VM OVFs in Data Center Lok >2014-04-21 17:36:05,648 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated VM OVFs in Data Center Lok >2014-04-21 17:36:05,648 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update template OVFs in Data Center Lok >2014-04-21 17:36:05,648 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated templates OVFs in Data Center Lok >2014-04-21 17:36:05,648 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 17:36:05,649 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 17:41:56,279 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2ebb2762 >2014-04-21 17:41:56,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 116aae6 >2014-04-21 17:41:56,505 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 116aae6 >2014-04-21 17:41:56,505 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2ebb2762 >2014-04-21 17:41:56,510 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 18:36:04,618 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-2) Start refreshing all users data >2014-04-21 18:36:05,649 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VMs/Templates Ovf. >2014-04-21 18:36:05,650 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update VM OVFs in Data Center Lok >2014-04-21 18:36:05,651 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated VM OVFs in Data Center Lok >2014-04-21 18:36:05,651 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to update template OVFs in Data Center Lok >2014-04-21 18:36:05,651 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully updated templates OVFs in Data Center Lok >2014-04-21 18:36:05,652 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 18:36:05,652 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-57) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 18:41:56,515 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7d9c9eb >2014-04-21 18:41:56,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6b576753 >2014-04-21 18:41:56,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6b576753 >2014-04-21 18:41:56,738 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7d9c9eb >2014-04-21 18:41:56,742 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 19:36:04,619 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-04-21 19:36:05,652 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-21 19:36:05,653 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-21 19:36:05,654 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-21 19:36:05,654 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-21 19:36:05,654 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-21 19:36:05,654 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 19:36:05,655 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 19:41:56,745 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 312d6299 >2014-04-21 19:41:56,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1aa44408 >2014-04-21 19:41:56,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1aa44408 >2014-04-21 19:41:56,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 312d6299 >2014-04-21 19:41:56,974 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 20:36:04,622 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-2) Start refreshing all users data >2014-04-21 20:36:05,655 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-21 20:36:05,656 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-21 20:36:05,657 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-21 20:36:05,657 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-21 20:36:05,657 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-21 20:36:05,657 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 20:36:05,658 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 20:41:56,978 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 197a6d85 >2014-04-21 20:41:56,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 34ac43d4 >2014-04-21 20:41:57,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 34ac43d4 >2014-04-21 20:41:57,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 197a6d85 >2014-04-21 20:41:57,209 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 21:36:04,625 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-04-21 21:36:05,658 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-04-21 21:36:05,659 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VM OVFs in Data Center Lok >2014-04-21 21:36:05,660 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated VM OVFs in Data Center Lok >2014-04-21 21:36:05,660 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update template OVFs in Data Center Lok >2014-04-21 21:36:05,660 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated templates OVFs in Data Center Lok >2014-04-21 21:36:05,660 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 21:36:05,661 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 21:41:57,212 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2f3af48a >2014-04-21 21:41:57,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 478ca83f >2014-04-21 21:41:57,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 478ca83f >2014-04-21 21:41:57,438 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2f3af48a >2014-04-21 21:41:57,443 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 22:36:04,626 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-04-21 22:36:05,661 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-04-21 22:36:05,662 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VM OVFs in Data Center Lok >2014-04-21 22:36:05,663 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully updated VM OVFs in Data Center Lok >2014-04-21 22:36:05,663 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update template OVFs in Data Center Lok >2014-04-21 22:36:05,664 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully updated templates OVFs in Data Center Lok >2014-04-21 22:36:05,664 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 22:36:05,664 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 22:41:57,446 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 34830982 >2014-04-21 22:41:57,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6e550bd >2014-04-21 22:41:57,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6e550bd >2014-04-21 22:41:57,672 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 34830982 >2014-04-21 22:41:57,677 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-21 23:36:04,627 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-04-21 23:36:05,664 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update VMs/Templates Ovf. >2014-04-21 23:36:05,665 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update VM OVFs in Data Center Lok >2014-04-21 23:36:05,666 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully updated VM OVFs in Data Center Lok >2014-04-21 23:36:05,666 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update template OVFs in Data Center Lok >2014-04-21 23:36:05,667 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully updated templates OVFs in Data Center Lok >2014-04-21 23:36:05,667 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-21 23:36:05,667 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-21 23:41:57,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7b774531 >2014-04-21 23:41:57,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 535fd34a >2014-04-21 23:41:57,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 535fd34a >2014-04-21 23:41:57,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7b774531 >2014-04-21 23:41:57,916 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 00:36:04,629 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-04-22 00:36:05,667 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VMs/Templates Ovf. >2014-04-22 00:36:05,668 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update VM OVFs in Data Center Lok >2014-04-22 00:36:05,669 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated VM OVFs in Data Center Lok >2014-04-22 00:36:05,669 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to update template OVFs in Data Center Lok >2014-04-22 00:36:05,669 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully updated templates OVFs in Data Center Lok >2014-04-22 00:36:05,669 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 00:36:05,670 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-63) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 00:41:57,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5c1ea589 >2014-04-22 00:41:57,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6005f49e >2014-04-22 00:41:58,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6005f49e >2014-04-22 00:41:58,153 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5c1ea589 >2014-04-22 00:41:58,158 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 01:36:04,631 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) Start refreshing all users data >2014-04-22 01:36:05,670 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-04-22 01:36:05,671 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VM OVFs in Data Center Lok >2014-04-22 01:36:05,672 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully updated VM OVFs in Data Center Lok >2014-04-22 01:36:05,672 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update template OVFs in Data Center Lok >2014-04-22 01:36:05,673 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully updated templates OVFs in Data Center Lok >2014-04-22 01:36:05,673 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 01:36:05,673 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 01:41:58,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 54640c5b >2014-04-22 01:41:58,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 746755a5 >2014-04-22 01:41:58,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 746755a5 >2014-04-22 01:41:58,390 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 54640c5b >2014-04-22 01:41:58,396 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 02:36:04,632 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-04-22 02:36:05,674 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-04-22 02:36:05,674 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VM OVFs in Data Center Lok >2014-04-22 02:36:05,675 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated VM OVFs in Data Center Lok >2014-04-22 02:36:05,675 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update template OVFs in Data Center Lok >2014-04-22 02:36:05,676 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated templates OVFs in Data Center Lok >2014-04-22 02:36:05,676 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 02:36:05,676 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 02:41:58,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 51cecb57 >2014-04-22 02:41:58,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5f87e6b6 >2014-04-22 02:41:58,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5f87e6b6 >2014-04-22 02:41:58,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 51cecb57 >2014-04-22 02:41:58,633 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-91) Start deleteAgedOutAuditLogs >2014-04-22 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-91) Finished deleteAgedOutAuditLogs >2014-04-22 03:36:04,634 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-04-22 03:36:05,676 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-04-22 03:36:05,677 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Lok >2014-04-22 03:36:05,677 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Lok >2014-04-22 03:36:05,677 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Lok >2014-04-22 03:36:05,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated templates OVFs in Data Center Lok >2014-04-22 03:36:05,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 03:36:05,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 03:41:58,635 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2e56df34 >2014-04-22 03:41:58,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4957e6b0 >2014-04-22 03:41:58,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4957e6b0 >2014-04-22 03:41:58,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e56df34 >2014-04-22 03:41:58,870 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 04:36:04,636 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-04-22 04:36:05,678 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-04-22 04:36:05,679 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VM OVFs in Data Center Lok >2014-04-22 04:36:05,680 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated VM OVFs in Data Center Lok >2014-04-22 04:36:05,680 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update template OVFs in Data Center Lok >2014-04-22 04:36:05,681 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated templates OVFs in Data Center Lok >2014-04-22 04:36:05,681 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 04:36:05,681 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 04:41:58,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [6d63ed15] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7ca2310b >2014-04-22 04:41:58,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [6d63ed15] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7c420919 >2014-04-22 04:41:59,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [6d63ed15] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c420919 >2014-04-22 04:41:59,251 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [6d63ed15] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7ca2310b >2014-04-22 04:41:59,255 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) [6d63ed15] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 05:36:04,638 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-22 05:36:05,681 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-04-22 05:36:05,682 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VM OVFs in Data Center Lok >2014-04-22 05:36:05,683 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated VM OVFs in Data Center Lok >2014-04-22 05:36:05,683 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update template OVFs in Data Center Lok >2014-04-22 05:36:05,689 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated templates OVFs in Data Center Lok >2014-04-22 05:36:05,689 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 05:36:05,690 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 05:41:59,259 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3a4173bc >2014-04-22 05:41:59,260 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1f155985 >2014-04-22 05:41:59,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1f155985 >2014-04-22 05:41:59,487 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3a4173bc >2014-04-22 05:41:59,492 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 06:36:04,640 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-04-22 06:36:05,691 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-04-22 06:36:05,692 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VM OVFs in Data Center Lok >2014-04-22 06:36:05,692 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated VM OVFs in Data Center Lok >2014-04-22 06:36:05,692 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update template OVFs in Data Center Lok >2014-04-22 06:36:05,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated templates OVFs in Data Center Lok >2014-04-22 06:36:05,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 06:36:05,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 06:41:59,516 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6a0ededb >2014-04-22 06:41:59,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 76493242 >2014-04-22 06:41:59,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76493242 >2014-04-22 06:41:59,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6a0ededb >2014-04-22 06:41:59,921 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 07:36:04,641 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-04-22 07:36:05,693 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-04-22 07:36:05,694 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VM OVFs in Data Center Lok >2014-04-22 07:36:05,694 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated VM OVFs in Data Center Lok >2014-04-22 07:36:05,694 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update template OVFs in Data Center Lok >2014-04-22 07:36:05,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated templates OVFs in Data Center Lok >2014-04-22 07:36:05,695 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 07:36:05,696 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 07:41:59,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 55d75982 >2014-04-22 07:41:59,924 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2d3591d5 >2014-04-22 07:42:00,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d3591d5 >2014-04-22 07:42:00,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 55d75982 >2014-04-22 07:42:00,154 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 08:36:04,643 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-22 08:36:05,696 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-04-22 08:36:05,697 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VM OVFs in Data Center Lok >2014-04-22 08:36:05,697 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated VM OVFs in Data Center Lok >2014-04-22 08:36:05,697 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update template OVFs in Data Center Lok >2014-04-22 08:36:05,698 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated templates OVFs in Data Center Lok >2014-04-22 08:36:05,698 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 08:36:05,699 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 08:42:00,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 498622dd >2014-04-22 08:42:00,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 55b00e1 >2014-04-22 08:42:00,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 55b00e1 >2014-04-22 08:42:00,389 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 498622dd >2014-04-22 08:42:00,393 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 09:36:04,644 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-04-22 09:36:05,700 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-22 09:36:05,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-22 09:36:05,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-22 09:36:05,701 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-22 09:36:05,702 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-22 09:36:05,702 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 09:36:05,702 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 09:42:00,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 49ff19ce >2014-04-22 09:42:00,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: af74956 >2014-04-22 09:42:00,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: af74956 >2014-04-22 09:42:00,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 49ff19ce >2014-04-22 09:42:00,651 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 10:36:04,646 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-04-22 10:36:05,702 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-04-22 10:36:05,703 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VM OVFs in Data Center Lok >2014-04-22 10:36:05,704 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated VM OVFs in Data Center Lok >2014-04-22 10:36:05,704 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update template OVFs in Data Center Lok >2014-04-22 10:36:05,705 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated templates OVFs in Data Center Lok >2014-04-22 10:36:05,705 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 10:36:05,705 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 10:42:00,654 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1ab94ed6 >2014-04-22 10:42:00,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7be6d14e >2014-04-22 10:42:00,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7be6d14e >2014-04-22 10:42:00,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1ab94ed6 >2014-04-22 10:42:00,887 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 11:36:04,648 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-04-22 11:36:05,706 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-04-22 11:36:05,707 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VM OVFs in Data Center Lok >2014-04-22 11:36:05,707 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated VM OVFs in Data Center Lok >2014-04-22 11:36:05,707 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update template OVFs in Data Center Lok >2014-04-22 11:36:05,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated templates OVFs in Data Center Lok >2014-04-22 11:36:05,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 11:36:05,708 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 11:42:00,891 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 78d39eb8 >2014-04-22 11:42:00,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: d598001 >2014-04-22 11:42:01,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: d598001 >2014-04-22 11:42:01,124 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 78d39eb8 >2014-04-22 11:42:01,128 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 12:36:04,651 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-04-22 12:36:05,709 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-22 12:36:05,710 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-22 12:36:05,711 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-22 12:36:05,711 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-22 12:36:05,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-22 12:36:05,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 12:36:05,712 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 12:42:01,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 104b9e83 >2014-04-22 12:42:01,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5493b2d0 >2014-04-22 12:42:01,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5493b2d0 >2014-04-22 12:42:01,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 104b9e83 >2014-04-22 12:42:01,361 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 13:36:04,652 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-13) Start refreshing all users data >2014-04-22 13:36:05,713 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-04-22 13:36:05,714 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VM OVFs in Data Center Lok >2014-04-22 13:36:05,714 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully updated VM OVFs in Data Center Lok >2014-04-22 13:36:05,714 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update template OVFs in Data Center Lok >2014-04-22 13:36:05,715 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully updated templates OVFs in Data Center Lok >2014-04-22 13:36:05,715 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 13:36:05,715 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 13:42:01,365 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: e831512 >2014-04-22 13:42:01,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3fdc582f >2014-04-22 13:42:01,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3fdc582f >2014-04-22 13:42:01,691 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: e831512 >2014-04-22 13:42:01,695 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 14:36:04,654 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-69) Start refreshing all users data >2014-04-22 14:36:05,716 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-04-22 14:36:05,717 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VM OVFs in Data Center Lok >2014-04-22 14:36:05,717 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated VM OVFs in Data Center Lok >2014-04-22 14:36:05,717 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update template OVFs in Data Center Lok >2014-04-22 14:36:05,718 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated templates OVFs in Data Center Lok >2014-04-22 14:36:05,718 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 14:36:05,718 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 14:42:01,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5e3ad5ec >2014-04-22 14:42:01,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 68b2a948 >2014-04-22 14:42:01,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 68b2a948 >2014-04-22 14:42:01,926 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e3ad5ec >2014-04-22 14:42:01,932 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) [35b64ebe] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 15:36:04,657 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-04-22 15:36:05,720 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-04-22 15:36:05,721 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VM OVFs in Data Center Lok >2014-04-22 15:36:05,722 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated VM OVFs in Data Center Lok >2014-04-22 15:36:05,722 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update template OVFs in Data Center Lok >2014-04-22 15:36:05,722 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated templates OVFs in Data Center Lok >2014-04-22 15:36:05,723 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 15:36:05,723 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 15:42:01,936 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7b060b14 >2014-04-22 15:42:01,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 49c18d25 >2014-04-22 15:42:02,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 49c18d25 >2014-04-22 15:42:02,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7b060b14 >2014-04-22 15:42:02,170 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 16:36:04,659 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-89) Start refreshing all users data >2014-04-22 16:36:05,723 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VMs/Templates Ovf. >2014-04-22 16:36:05,724 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update VM OVFs in Data Center Lok >2014-04-22 16:36:05,725 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated VM OVFs in Data Center Lok >2014-04-22 16:36:05,725 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to update template OVFs in Data Center Lok >2014-04-22 16:36:05,726 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully updated templates OVFs in Data Center Lok >2014-04-22 16:36:05,726 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 16:36:05,727 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-35) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 16:42:02,175 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 38b2f01d >2014-04-22 16:42:02,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6bb21cff >2014-04-22 16:42:02,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6bb21cff >2014-04-22 16:42:02,400 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 38b2f01d >2014-04-22 16:42:02,405 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 17:36:04,660 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-22 17:36:05,727 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-22 17:36:05,728 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-22 17:36:05,729 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-22 17:36:05,729 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-22 17:36:05,729 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-22 17:36:05,729 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 17:36:05,730 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 17:42:02,408 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 71ac74dd >2014-04-22 17:42:02,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 712c9378 >2014-04-22 17:42:02,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 712c9378 >2014-04-22 17:42:02,637 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 71ac74dd >2014-04-22 17:42:02,648 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 18:36:04,662 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-22 18:36:05,730 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-04-22 18:36:05,731 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VM OVFs in Data Center Lok >2014-04-22 18:36:05,732 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated VM OVFs in Data Center Lok >2014-04-22 18:36:05,732 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update template OVFs in Data Center Lok >2014-04-22 18:36:05,732 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated templates OVFs in Data Center Lok >2014-04-22 18:36:05,732 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 18:36:05,733 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 18:42:02,653 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1709fb49 >2014-04-22 18:42:02,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 10508170 >2014-04-22 18:42:02,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 10508170 >2014-04-22 18:42:02,876 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1709fb49 >2014-04-22 18:42:02,881 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 19:36:04,664 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-67) Start refreshing all users data >2014-04-22 19:36:05,734 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VMs/Templates Ovf. >2014-04-22 19:36:05,735 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update VM OVFs in Data Center Lok >2014-04-22 19:36:05,736 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated VM OVFs in Data Center Lok >2014-04-22 19:36:05,736 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to update template OVFs in Data Center Lok >2014-04-22 19:36:05,736 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully updated templates OVFs in Data Center Lok >2014-04-22 19:36:05,736 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 19:36:05,737 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 19:42:02,884 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4b021ddb >2014-04-22 19:42:02,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 77f63b55 >2014-04-22 19:42:03,427 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 77f63b55 >2014-04-22 19:42:03,427 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4b021ddb >2014-04-22 19:42:03,431 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 20:36:04,666 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) Start refreshing all users data >2014-04-22 20:36:05,737 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-04-22 20:36:05,738 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VM OVFs in Data Center Lok >2014-04-22 20:36:05,739 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated VM OVFs in Data Center Lok >2014-04-22 20:36:05,739 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update template OVFs in Data Center Lok >2014-04-22 20:36:05,739 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated templates OVFs in Data Center Lok >2014-04-22 20:36:05,739 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 20:36:05,740 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 20:42:03,435 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 161e73c0 >2014-04-22 20:42:03,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 29414d98 >2014-04-22 20:42:03,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 29414d98 >2014-04-22 20:42:03,826 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 161e73c0 >2014-04-22 20:42:03,831 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 21:36:04,668 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-78) Start refreshing all users data >2014-04-22 21:36:05,742 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-04-22 21:36:05,743 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VM OVFs in Data Center Lok >2014-04-22 21:36:05,744 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated VM OVFs in Data Center Lok >2014-04-22 21:36:05,744 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update template OVFs in Data Center Lok >2014-04-22 21:36:05,745 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated templates OVFs in Data Center Lok >2014-04-22 21:36:05,745 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 21:36:05,745 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 21:42:03,834 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 26ce1798 >2014-04-22 21:42:03,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 246f100f >2014-04-22 21:42:04,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 246f100f >2014-04-22 21:42:04,058 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 26ce1798 >2014-04-22 21:42:04,063 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 22:36:04,671 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-59) Start refreshing all users data >2014-04-22 22:36:05,745 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update VMs/Templates Ovf. >2014-04-22 22:36:05,746 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update VM OVFs in Data Center Lok >2014-04-22 22:36:05,747 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully updated VM OVFs in Data Center Lok >2014-04-22 22:36:05,747 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update template OVFs in Data Center Lok >2014-04-22 22:36:05,747 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully updated templates OVFs in Data Center Lok >2014-04-22 22:36:05,747 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 22:36:05,748 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 22:42:04,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4e890b7f >2014-04-22 22:42:04,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 751e721 >2014-04-22 22:42:04,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 751e721 >2014-04-22 22:42:04,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4e890b7f >2014-04-22 22:42:04,301 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-22 23:36:04,672 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-04-22 23:36:05,748 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-04-22 23:36:05,749 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VM OVFs in Data Center Lok >2014-04-22 23:36:05,750 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated VM OVFs in Data Center Lok >2014-04-22 23:36:05,750 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update template OVFs in Data Center Lok >2014-04-22 23:36:05,751 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated templates OVFs in Data Center Lok >2014-04-22 23:36:05,751 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-22 23:36:05,751 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-22 23:42:04,304 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 76d6eeb7 >2014-04-22 23:42:04,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 18d26d55 >2014-04-22 23:42:04,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 18d26d55 >2014-04-22 23:42:04,527 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 76d6eeb7 >2014-04-22 23:42:04,553 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 00:36:04,673 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-67) Start refreshing all users data >2014-04-23 00:36:05,751 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-04-23 00:36:05,752 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VM OVFs in Data Center Lok >2014-04-23 00:36:05,753 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully updated VM OVFs in Data Center Lok >2014-04-23 00:36:05,753 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update template OVFs in Data Center Lok >2014-04-23 00:36:05,754 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully updated templates OVFs in Data Center Lok >2014-04-23 00:36:05,754 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 00:36:05,754 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 00:42:04,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5e66a43d >2014-04-23 00:42:04,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 24a8d136 >2014-04-23 00:42:04,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 24a8d136 >2014-04-23 00:42:04,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e66a43d >2014-04-23 00:42:04,783 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 01:36:04,675 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-34) Start refreshing all users data >2014-04-23 01:36:05,754 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-04-23 01:36:05,755 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VM OVFs in Data Center Lok >2014-04-23 01:36:05,756 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated VM OVFs in Data Center Lok >2014-04-23 01:36:05,756 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update template OVFs in Data Center Lok >2014-04-23 01:36:05,756 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated templates OVFs in Data Center Lok >2014-04-23 01:36:05,756 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 01:36:05,757 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 01:42:04,788 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 19c5e3fa >2014-04-23 01:42:04,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: efa3f4e >2014-04-23 01:42:05,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: efa3f4e >2014-04-23 01:42:05,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 19c5e3fa >2014-04-23 01:42:05,191 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 02:36:04,677 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-04-23 02:36:05,758 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-23 02:36:05,758 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-23 02:36:05,759 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-23 02:36:05,759 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-23 02:36:05,760 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-23 02:36:05,760 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 02:36:05,760 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 02:42:05,195 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6e00ae64 >2014-04-23 02:42:05,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6eb95fa7 >2014-04-23 02:42:05,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6eb95fa7 >2014-04-23 02:42:05,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6e00ae64 >2014-04-23 02:42:05,425 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-47) Start deleteAgedOutAuditLogs >2014-04-23 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-47) Finished deleteAgedOutAuditLogs >2014-04-23 03:36:04,679 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-04-23 03:36:05,760 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update VMs/Templates Ovf. >2014-04-23 03:36:05,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update VM OVFs in Data Center Lok >2014-04-23 03:36:05,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully updated VM OVFs in Data Center Lok >2014-04-23 03:36:05,762 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update template OVFs in Data Center Lok >2014-04-23 03:36:05,763 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully updated templates OVFs in Data Center Lok >2014-04-23 03:36:05,763 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 03:36:05,764 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 03:42:05,430 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 48af6465 >2014-04-23 03:42:05,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 72e8482a >2014-04-23 03:42:05,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 72e8482a >2014-04-23 03:42:05,658 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 48af6465 >2014-04-23 03:42:05,663 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 04:36:04,680 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-04-23 04:36:05,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-23 04:36:05,765 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-23 04:36:05,766 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-23 04:36:05,766 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-23 04:36:05,767 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-23 04:36:05,767 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 04:36:05,767 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 04:42:05,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2a6db311 >2014-04-23 04:42:05,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 80736da >2014-04-23 04:42:05,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 80736da >2014-04-23 04:42:05,886 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2a6db311 >2014-04-23 04:42:05,890 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 05:36:04,682 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-04-23 05:36:05,767 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-23 05:36:05,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-23 05:36:05,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-23 05:36:05,768 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-23 05:36:05,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-23 05:36:05,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 05:36:05,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 05:42:05,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4ec68052 >2014-04-23 05:42:06,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 325b0c8e >2014-04-23 05:42:06,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 325b0c8e >2014-04-23 05:42:06,528 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4ec68052 >2014-04-23 05:42:06,533 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 06:36:04,684 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-23) Start refreshing all users data >2014-04-23 06:36:05,769 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-04-23 06:36:05,770 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VM OVFs in Data Center Lok >2014-04-23 06:36:05,771 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated VM OVFs in Data Center Lok >2014-04-23 06:36:05,771 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update template OVFs in Data Center Lok >2014-04-23 06:36:05,771 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated templates OVFs in Data Center Lok >2014-04-23 06:36:05,771 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 06:36:05,772 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 06:42:06,539 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7e1947aa >2014-04-23 06:42:06,540 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2db02a93 >2014-04-23 06:42:07,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2db02a93 >2014-04-23 06:42:07,056 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7e1947aa >2014-04-23 06:42:07,060 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 07:36:04,686 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-04-23 07:36:05,772 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-04-23 07:36:05,773 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VM OVFs in Data Center Lok >2014-04-23 07:36:05,774 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated VM OVFs in Data Center Lok >2014-04-23 07:36:05,774 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update template OVFs in Data Center Lok >2014-04-23 07:36:05,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated templates OVFs in Data Center Lok >2014-04-23 07:36:05,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 07:36:05,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 07:42:07,063 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 58c9554e >2014-04-23 07:42:07,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 9ffa16c >2014-04-23 07:42:07,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 9ffa16c >2014-04-23 07:42:07,454 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 58c9554e >2014-04-23 07:42:07,460 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 08:36:04,688 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-23 08:36:05,775 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update VMs/Templates Ovf. >2014-04-23 08:36:05,777 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update VM OVFs in Data Center Lok >2014-04-23 08:36:05,777 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully updated VM OVFs in Data Center Lok >2014-04-23 08:36:05,777 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to update template OVFs in Data Center Lok >2014-04-23 08:36:05,778 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully updated templates OVFs in Data Center Lok >2014-04-23 08:36:05,778 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 08:36:05,779 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 08:42:07,462 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 635c6819 >2014-04-23 08:42:07,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 47b67445 >2014-04-23 08:42:07,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 47b67445 >2014-04-23 08:42:07,690 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 635c6819 >2014-04-23 08:42:07,694 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 09:36:04,690 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-04-23 09:36:05,779 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VMs/Templates Ovf. >2014-04-23 09:36:05,780 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update VM OVFs in Data Center Lok >2014-04-23 09:36:05,781 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully updated VM OVFs in Data Center Lok >2014-04-23 09:36:05,781 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to update template OVFs in Data Center Lok >2014-04-23 09:36:05,782 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully updated templates OVFs in Data Center Lok >2014-04-23 09:36:05,782 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 09:36:05,782 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-96) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 09:42:07,698 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: ccf7a04 >2014-04-23 09:42:07,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 953a32e >2014-04-23 09:42:08,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 953a32e >2014-04-23 09:42:08,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ccf7a04 >2014-04-23 09:42:08,023 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 10:36:04,692 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-1) Start refreshing all users data >2014-04-23 10:36:05,783 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VMs/Templates Ovf. >2014-04-23 10:36:05,784 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update VM OVFs in Data Center Lok >2014-04-23 10:36:05,785 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated VM OVFs in Data Center Lok >2014-04-23 10:36:05,785 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to update template OVFs in Data Center Lok >2014-04-23 10:36:05,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully updated templates OVFs in Data Center Lok >2014-04-23 10:36:05,786 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 10:36:05,787 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-64) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 10:42:08,028 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3ecd8626 >2014-04-23 10:42:08,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 27224900 >2014-04-23 10:42:08,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 27224900 >2014-04-23 10:42:08,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3ecd8626 >2014-04-23 10:42:08,569 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 11:36:04,693 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) Start refreshing all users data >2014-04-23 11:36:05,788 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-04-23 11:36:05,788 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VM OVFs in Data Center Lok >2014-04-23 11:36:05,789 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated VM OVFs in Data Center Lok >2014-04-23 11:36:05,789 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update template OVFs in Data Center Lok >2014-04-23 11:36:05,790 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated templates OVFs in Data Center Lok >2014-04-23 11:36:05,790 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 11:36:05,790 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 11:42:08,573 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5c8ff5f1 >2014-04-23 11:42:08,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6f5bd0b0 >2014-04-23 11:42:08,802 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6f5bd0b0 >2014-04-23 11:42:08,802 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5c8ff5f1 >2014-04-23 11:42:08,808 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 12:36:04,695 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-04-23 12:36:05,790 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-23 12:36:05,791 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-23 12:36:05,792 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-23 12:36:05,792 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-23 12:36:05,793 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-23 12:36:05,793 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 12:36:05,793 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 12:42:08,813 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5c08c869 >2014-04-23 12:42:08,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 449d94b3 >2014-04-23 12:42:09,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 449d94b3 >2014-04-23 12:42:09,043 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5c08c869 >2014-04-23 12:42:09,048 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 13:36:04,697 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-04-23 13:36:05,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VMs/Templates Ovf. >2014-04-23 13:36:05,794 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update VM OVFs in Data Center Lok >2014-04-23 13:36:05,795 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Successfully updated VM OVFs in Data Center Lok >2014-04-23 13:36:05,795 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to update template OVFs in Data Center Lok >2014-04-23 13:36:05,796 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Successfully updated templates OVFs in Data Center Lok >2014-04-23 13:36:05,796 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 13:36:05,796 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-71) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 13:42:09,051 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4ddaf517 >2014-04-23 13:42:09,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6210e0ef >2014-04-23 13:42:09,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6210e0ef >2014-04-23 13:42:09,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4ddaf517 >2014-04-23 13:42:09,289 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 14:36:04,699 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-04-23 14:36:05,796 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-23 14:36:05,797 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-23 14:36:05,798 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-23 14:36:05,798 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-23 14:36:05,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-23 14:36:05,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 14:36:05,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 14:42:09,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 35d05213 >2014-04-23 14:42:09,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7fbc78fe >2014-04-23 14:42:09,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7fbc78fe >2014-04-23 14:42:09,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 35d05213 >2014-04-23 14:42:09,526 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 15:36:04,701 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-23 15:36:05,799 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VMs/Templates Ovf. >2014-04-23 15:36:05,800 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update VM OVFs in Data Center Lok >2014-04-23 15:36:05,801 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated VM OVFs in Data Center Lok >2014-04-23 15:36:05,801 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to update template OVFs in Data Center Lok >2014-04-23 15:36:05,801 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully updated templates OVFs in Data Center Lok >2014-04-23 15:36:05,801 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 15:36:05,802 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 15:42:09,532 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7c0c3bf8 >2014-04-23 15:42:09,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7355668 >2014-04-23 15:42:09,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7355668 >2014-04-23 15:42:09,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c0c3bf8 >2014-04-23 15:42:09,762 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 16:36:04,702 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-23 16:36:05,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VMs/Templates Ovf. >2014-04-23 16:36:05,803 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update VM OVFs in Data Center Lok >2014-04-23 16:36:05,804 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated VM OVFs in Data Center Lok >2014-04-23 16:36:05,804 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to update template OVFs in Data Center Lok >2014-04-23 16:36:05,805 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully updated templates OVFs in Data Center Lok >2014-04-23 16:36:05,805 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 16:36:05,805 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-9) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 16:42:09,766 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 724c555e >2014-04-23 16:42:09,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7178c8f1 >2014-04-23 16:42:09,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7178c8f1 >2014-04-23 16:42:09,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 724c555e >2014-04-23 16:42:09,999 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 17:36:04,705 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-04-23 17:36:05,806 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VMs/Templates Ovf. >2014-04-23 17:36:05,806 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update VM OVFs in Data Center Lok >2014-04-23 17:36:05,807 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated VM OVFs in Data Center Lok >2014-04-23 17:36:05,807 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to update template OVFs in Data Center Lok >2014-04-23 17:36:05,808 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully updated templates OVFs in Data Center Lok >2014-04-23 17:36:05,808 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 17:36:05,808 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-77) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 17:42:10,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7332866f >2014-04-23 17:42:10,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7a181c7e >2014-04-23 17:42:10,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7a181c7e >2014-04-23 17:42:10,230 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7332866f >2014-04-23 17:42:10,234 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 18:36:04,706 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-04-23 18:36:05,809 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VMs/Templates Ovf. >2014-04-23 18:36:05,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update VM OVFs in Data Center Lok >2014-04-23 18:36:05,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated VM OVFs in Data Center Lok >2014-04-23 18:36:05,810 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to update template OVFs in Data Center Lok >2014-04-23 18:36:05,811 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully updated templates OVFs in Data Center Lok >2014-04-23 18:36:05,811 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 18:36:05,812 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-11) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 18:42:10,237 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5ffca973 >2014-04-23 18:42:10,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 44d8ee40 >2014-04-23 18:42:10,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 44d8ee40 >2014-04-23 18:42:10,467 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5ffca973 >2014-04-23 18:42:10,475 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) [35b64ebe] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 19:36:04,708 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-23 19:36:05,813 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-04-23 19:36:05,813 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-04-23 19:36:05,814 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-04-23 19:36:05,814 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-04-23 19:36:05,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-04-23 19:36:05,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 19:36:05,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 19:42:10,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5480d7f7 >2014-04-23 19:42:10,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 56e2c478 >2014-04-23 19:42:10,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 56e2c478 >2014-04-23 19:42:10,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5480d7f7 >2014-04-23 19:42:10,952 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 20:36:04,710 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-18) Start refreshing all users data >2014-04-23 20:36:05,815 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-04-23 20:36:05,816 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VM OVFs in Data Center Lok >2014-04-23 20:36:05,817 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated VM OVFs in Data Center Lok >2014-04-23 20:36:05,817 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update template OVFs in Data Center Lok >2014-04-23 20:36:05,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated templates OVFs in Data Center Lok >2014-04-23 20:36:05,818 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 20:36:05,819 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 20:42:10,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: c6ffa31 >2014-04-23 20:42:10,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: b92ad58 >2014-04-23 20:42:11,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: b92ad58 >2014-04-23 20:42:11,184 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: c6ffa31 >2014-04-23 20:42:11,188 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 21:36:04,712 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-23 21:36:05,819 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-23 21:36:05,820 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-23 21:36:05,821 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-23 21:36:05,821 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-23 21:36:05,822 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-23 21:36:05,822 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 21:36:05,822 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 21:42:11,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 636293cd >2014-04-23 21:42:11,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5cd5aab4 >2014-04-23 21:42:11,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5cd5aab4 >2014-04-23 21:42:11,423 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 636293cd >2014-04-23 21:42:11,428 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 22:36:04,713 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-04-23 22:36:05,822 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-04-23 22:36:05,823 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VM OVFs in Data Center Lok >2014-04-23 22:36:05,824 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated VM OVFs in Data Center Lok >2014-04-23 22:36:05,824 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update template OVFs in Data Center Lok >2014-04-23 22:36:05,825 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated templates OVFs in Data Center Lok >2014-04-23 22:36:05,825 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 22:36:05,826 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 22:42:11,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 97e1962 >2014-04-23 22:42:11,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 35b475cc >2014-04-23 22:42:11,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 35b475cc >2014-04-23 22:42:11,662 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 97e1962 >2014-04-23 22:42:11,666 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-23 23:36:04,715 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-04-23 23:36:05,826 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-04-23 23:36:05,827 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VM OVFs in Data Center Lok >2014-04-23 23:36:05,828 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated VM OVFs in Data Center Lok >2014-04-23 23:36:05,828 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update template OVFs in Data Center Lok >2014-04-23 23:36:05,829 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated templates OVFs in Data Center Lok >2014-04-23 23:36:05,829 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-23 23:36:05,829 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-23 23:42:11,669 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4a0e3c80 >2014-04-23 23:42:11,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6ca61d7b >2014-04-23 23:42:11,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ca61d7b >2014-04-23 23:42:11,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4a0e3c80 >2014-04-23 23:42:11,926 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 00:36:04,716 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) Start refreshing all users data >2014-04-24 00:36:05,829 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-24 00:36:05,830 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-24 00:36:05,831 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-24 00:36:05,831 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-24 00:36:05,831 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-24 00:36:05,831 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 00:36:05,832 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 00:42:11,929 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1b51655c >2014-04-24 00:42:11,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7cd58906 >2014-04-24 00:42:12,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7cd58906 >2014-04-24 00:42:12,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b51655c >2014-04-24 00:42:12,162 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 01:36:04,718 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-63) Start refreshing all users data >2014-04-24 01:36:05,832 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-04-24 01:36:05,833 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VM OVFs in Data Center Lok >2014-04-24 01:36:05,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated VM OVFs in Data Center Lok >2014-04-24 01:36:05,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update template OVFs in Data Center Lok >2014-04-24 01:36:05,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated templates OVFs in Data Center Lok >2014-04-24 01:36:05,834 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 01:36:05,835 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 01:42:12,165 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4bd15a73 >2014-04-24 01:42:12,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1d895ae8 >2014-04-24 01:42:12,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d895ae8 >2014-04-24 01:42:12,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4bd15a73 >2014-04-24 01:42:12,400 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 02:36:04,720 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-04-24 02:36:05,836 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-04-24 02:36:05,836 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VM OVFs in Data Center Lok >2014-04-24 02:36:05,837 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated VM OVFs in Data Center Lok >2014-04-24 02:36:05,837 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update template OVFs in Data Center Lok >2014-04-24 02:36:05,838 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated templates OVFs in Data Center Lok >2014-04-24 02:36:05,838 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 02:36:05,838 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 02:42:12,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2cbbd0ed >2014-04-24 02:42:12,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5e918e8c >2014-04-24 02:42:12,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5e918e8c >2014-04-24 02:42:12,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2cbbd0ed >2014-04-24 02:42:12,635 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Start deleteAgedOutAuditLogs >2014-04-24 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Finished deleteAgedOutAuditLogs >2014-04-24 03:36:04,721 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-24 03:36:05,838 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-24 03:36:05,839 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-04-24 03:36:05,840 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-04-24 03:36:05,840 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-04-24 03:36:05,840 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-04-24 03:36:05,840 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 03:36:05,841 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 03:42:12,639 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: ae7d7b9 >2014-04-24 03:42:12,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7b03b846 >2014-04-24 03:42:12,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7b03b846 >2014-04-24 03:42:12,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ae7d7b9 >2014-04-24 03:42:12,872 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 04:36:04,722 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-80) Start refreshing all users data >2014-04-24 04:36:05,842 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-04-24 04:36:05,843 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VM OVFs in Data Center Lok >2014-04-24 04:36:05,844 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated VM OVFs in Data Center Lok >2014-04-24 04:36:05,844 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update template OVFs in Data Center Lok >2014-04-24 04:36:05,845 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated templates OVFs in Data Center Lok >2014-04-24 04:36:05,845 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 04:36:05,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 04:42:12,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1fa63d2b >2014-04-24 04:42:13,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2804ae63 >2014-04-24 04:42:13,285 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2804ae63 >2014-04-24 04:42:13,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1fa63d2b >2014-04-24 04:42:13,290 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 05:36:04,724 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-6) Start refreshing all users data >2014-04-24 05:36:05,846 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-04-24 05:36:05,847 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Lok >2014-04-24 05:36:05,847 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Lok >2014-04-24 05:36:05,847 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Lok >2014-04-24 05:36:05,848 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center Lok >2014-04-24 05:36:05,848 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 05:36:05,849 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 05:42:13,295 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6a4ffeed >2014-04-24 05:42:13,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 29e9b022 >2014-04-24 05:42:13,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 29e9b022 >2014-04-24 05:42:13,522 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6a4ffeed >2014-04-24 05:42:13,527 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 06:36:04,725 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-25) Start refreshing all users data >2014-04-24 06:36:05,850 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-04-24 06:36:05,850 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VM OVFs in Data Center Lok >2014-04-24 06:36:05,851 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated VM OVFs in Data Center Lok >2014-04-24 06:36:05,851 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update template OVFs in Data Center Lok >2014-04-24 06:36:05,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated templates OVFs in Data Center Lok >2014-04-24 06:36:05,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 06:36:05,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 06:42:13,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3e0c7577 >2014-04-24 06:42:13,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 23f3f774 >2014-04-24 06:42:13,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23f3f774 >2014-04-24 06:42:13,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3e0c7577 >2014-04-24 06:42:13,759 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 07:36:04,727 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-61) Start refreshing all users data >2014-04-24 07:36:05,852 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VMs/Templates Ovf. >2014-04-24 07:36:05,854 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update VM OVFs in Data Center Lok >2014-04-24 07:36:05,855 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated VM OVFs in Data Center Lok >2014-04-24 07:36:05,855 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to update template OVFs in Data Center Lok >2014-04-24 07:36:05,856 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully updated templates OVFs in Data Center Lok >2014-04-24 07:36:05,856 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 07:36:05,857 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-18) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 07:42:13,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7d8b2d56 >2014-04-24 07:42:13,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3c4da97e >2014-04-24 07:42:13,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c4da97e >2014-04-24 07:42:13,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7d8b2d56 >2014-04-24 07:42:13,993 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 08:36:04,728 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-04-24 08:36:05,857 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VMs/Templates Ovf. >2014-04-24 08:36:05,858 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update VM OVFs in Data Center Lok >2014-04-24 08:36:05,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated VM OVFs in Data Center Lok >2014-04-24 08:36:05,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to update template OVFs in Data Center Lok >2014-04-24 08:36:05,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully updated templates OVFs in Data Center Lok >2014-04-24 08:36:05,860 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 08:36:05,861 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-67) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 08:42:13,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4e9bbe57 >2014-04-24 08:42:13,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 52ed6a12 >2014-04-24 08:42:14,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 52ed6a12 >2014-04-24 08:42:14,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4e9bbe57 >2014-04-24 08:42:14,231 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 09:36:04,730 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-68) Start refreshing all users data >2014-04-24 09:36:05,862 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-04-24 09:36:05,863 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VM OVFs in Data Center Lok >2014-04-24 09:36:05,863 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated VM OVFs in Data Center Lok >2014-04-24 09:36:05,863 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update template OVFs in Data Center Lok >2014-04-24 09:36:05,864 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated templates OVFs in Data Center Lok >2014-04-24 09:36:05,864 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 09:36:05,864 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 09:42:14,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) [35b64ebe] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1bd0971d >2014-04-24 09:42:14,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) [35b64ebe] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1b70ff2f >2014-04-24 09:42:14,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) [35b64ebe] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b70ff2f >2014-04-24 09:42:14,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) [35b64ebe] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1bd0971d >2014-04-24 09:42:14,468 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) [35b64ebe] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 10:36:04,733 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-04-24 10:36:05,866 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-04-24 10:36:05,866 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VM OVFs in Data Center Lok >2014-04-24 10:36:05,867 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated VM OVFs in Data Center Lok >2014-04-24 10:36:05,867 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update template OVFs in Data Center Lok >2014-04-24 10:36:05,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated templates OVFs in Data Center Lok >2014-04-24 10:36:05,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 10:36:05,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 10:42:14,470 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 12416d21 >2014-04-24 10:42:14,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5639ae4 >2014-04-24 10:42:14,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5639ae4 >2014-04-24 10:42:14,706 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 12416d21 >2014-04-24 10:42:14,711 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 11:36:04,734 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) Start refreshing all users data >2014-04-24 11:36:05,868 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-04-24 11:36:05,869 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VM OVFs in Data Center Lok >2014-04-24 11:36:05,869 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated VM OVFs in Data Center Lok >2014-04-24 11:36:05,870 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update template OVFs in Data Center Lok >2014-04-24 11:36:05,870 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated templates OVFs in Data Center Lok >2014-04-24 11:36:05,870 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 11:36:05,871 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 11:42:14,714 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: eef98de >2014-04-24 11:42:14,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 141d1e7b >2014-04-24 11:42:14,960 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 141d1e7b >2014-04-24 11:42:14,960 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: eef98de >2014-04-24 11:42:14,965 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 12:36:04,736 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-69) Start refreshing all users data >2014-04-24 12:36:05,871 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-24 12:36:05,872 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-24 12:36:05,872 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-24 12:36:05,872 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-24 12:36:05,873 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-24 12:36:05,873 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 12:36:05,874 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 12:42:14,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5cd65fab >2014-04-24 12:42:14,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7e23032a >2014-04-24 12:42:15,193 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7e23032a >2014-04-24 12:42:15,193 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5cd65fab >2014-04-24 12:42:15,198 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 13:36:04,739 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-24 13:36:05,874 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VMs/Templates Ovf. >2014-04-24 13:36:05,875 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update VM OVFs in Data Center Lok >2014-04-24 13:36:05,875 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated VM OVFs in Data Center Lok >2014-04-24 13:36:05,875 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to update template OVFs in Data Center Lok >2014-04-24 13:36:05,876 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully updated templates OVFs in Data Center Lok >2014-04-24 13:36:05,876 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 13:36:05,876 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-61) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 13:42:15,202 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3c87e90a >2014-04-24 13:42:15,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7ed4ae91 >2014-04-24 13:42:15,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7ed4ae91 >2014-04-24 13:42:15,431 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3c87e90a >2014-04-24 13:42:15,436 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 14:36:04,741 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-63) Start refreshing all users data >2014-04-24 14:36:05,876 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VMs/Templates Ovf. >2014-04-24 14:36:05,877 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update VM OVFs in Data Center Lok >2014-04-24 14:36:05,877 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated VM OVFs in Data Center Lok >2014-04-24 14:36:05,877 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to update template OVFs in Data Center Lok >2014-04-24 14:36:05,878 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully updated templates OVFs in Data Center Lok >2014-04-24 14:36:05,878 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 14:36:05,878 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-52) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 14:42:15,439 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 80e3f41 >2014-04-24 14:42:15,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5f5ac888 >2014-04-24 14:42:15,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5f5ac888 >2014-04-24 14:42:15,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 80e3f41 >2014-04-24 14:42:15,675 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 15:36:04,744 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-24 15:36:05,879 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VMs/Templates Ovf. >2014-04-24 15:36:05,879 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update VM OVFs in Data Center Lok >2014-04-24 15:36:05,880 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated VM OVFs in Data Center Lok >2014-04-24 15:36:05,880 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to update template OVFs in Data Center Lok >2014-04-24 15:36:05,880 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully updated templates OVFs in Data Center Lok >2014-04-24 15:36:05,880 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 15:36:05,881 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-1) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 15:42:15,679 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7d7b7fce >2014-04-24 15:42:15,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3703edbe >2014-04-24 15:42:15,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3703edbe >2014-04-24 15:42:15,911 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7d7b7fce >2014-04-24 15:42:15,916 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 16:36:04,748 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-24 16:36:05,882 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-24 16:36:05,882 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-24 16:36:05,883 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-24 16:36:05,883 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-24 16:36:05,884 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-24 16:36:05,884 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 16:36:05,884 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 16:42:15,921 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) [431634a] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1742803b >2014-04-24 16:42:15,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) [431634a] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: ef0ad6b >2014-04-24 16:42:16,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) [431634a] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ef0ad6b >2014-04-24 16:42:16,321 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) [431634a] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1742803b >2014-04-24 16:42:16,325 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) [431634a] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 17:36:04,748 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-04-24 17:36:05,884 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-24 17:36:05,885 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-04-24 17:36:05,886 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-04-24 17:36:05,886 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-04-24 17:36:05,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-04-24 17:36:05,887 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 17:36:05,888 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 17:42:16,329 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1c6ebddc >2014-04-24 17:42:16,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2a72c519 >2014-04-24 17:42:16,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2a72c519 >2014-04-24 17:42:16,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1c6ebddc >2014-04-24 17:42:16,587 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 18:36:04,751 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-25) Start refreshing all users data >2014-04-24 18:36:05,888 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-04-24 18:36:05,889 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VM OVFs in Data Center Lok >2014-04-24 18:36:05,890 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated VM OVFs in Data Center Lok >2014-04-24 18:36:05,890 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update template OVFs in Data Center Lok >2014-04-24 18:36:05,890 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated templates OVFs in Data Center Lok >2014-04-24 18:36:05,890 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 18:36:05,891 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 18:42:16,592 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1ff09ec5 >2014-04-24 18:42:16,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6e47a7e4 >2014-04-24 18:42:16,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6e47a7e4 >2014-04-24 18:42:16,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1ff09ec5 >2014-04-24 18:42:16,839 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 19:36:04,753 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-20) Start refreshing all users data >2014-04-24 19:36:05,892 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update VMs/Templates Ovf. >2014-04-24 19:36:05,892 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update VM OVFs in Data Center Lok >2014-04-24 19:36:05,893 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully updated VM OVFs in Data Center Lok >2014-04-24 19:36:05,893 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update template OVFs in Data Center Lok >2014-04-24 19:36:05,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully updated templates OVFs in Data Center Lok >2014-04-24 19:36:05,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 19:36:05,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 19:42:16,843 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: b2cb522 >2014-04-24 19:42:16,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2427d0e9 >2014-04-24 19:42:17,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2427d0e9 >2014-04-24 19:42:17,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: b2cb522 >2014-04-24 19:42:17,076 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 20:36:04,755 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-04-24 20:36:05,894 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-04-24 20:36:05,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VM OVFs in Data Center Lok >2014-04-24 20:36:05,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated VM OVFs in Data Center Lok >2014-04-24 20:36:05,895 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update template OVFs in Data Center Lok >2014-04-24 20:36:05,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated templates OVFs in Data Center Lok >2014-04-24 20:36:05,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 20:36:05,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 20:42:17,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5dedbe33 >2014-04-24 20:42:17,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 23623672 >2014-04-24 20:42:17,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 23623672 >2014-04-24 20:42:17,617 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5dedbe33 >2014-04-24 20:42:17,621 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 21:36:04,757 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-52) Start refreshing all users data >2014-04-24 21:36:05,896 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VMs/Templates Ovf. >2014-04-24 21:36:05,897 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update VM OVFs in Data Center Lok >2014-04-24 21:36:05,897 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated VM OVFs in Data Center Lok >2014-04-24 21:36:05,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to update template OVFs in Data Center Lok >2014-04-24 21:36:05,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully updated templates OVFs in Data Center Lok >2014-04-24 21:36:05,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 21:36:05,898 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-17) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 21:42:17,629 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7d8ecfe0 >2014-04-24 21:42:17,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2af1bb86 >2014-04-24 21:42:18,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2af1bb86 >2014-04-24 21:42:18,131 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7d8ecfe0 >2014-04-24 21:42:18,136 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 22:36:04,758 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-04-24 22:36:05,899 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VMs/Templates Ovf. >2014-04-24 22:36:05,899 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update VM OVFs in Data Center Lok >2014-04-24 22:36:05,900 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated VM OVFs in Data Center Lok >2014-04-24 22:36:05,900 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to update template OVFs in Data Center Lok >2014-04-24 22:36:05,900 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully updated templates OVFs in Data Center Lok >2014-04-24 22:36:05,900 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 22:36:05,901 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-6) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 22:42:18,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 59639d25 >2014-04-24 22:42:18,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1ae215fc >2014-04-24 22:42:18,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1ae215fc >2014-04-24 22:42:18,370 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 59639d25 >2014-04-24 22:42:18,375 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-24 23:36:04,760 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-24 23:36:05,901 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VMs/Templates Ovf. >2014-04-24 23:36:05,901 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update VM OVFs in Data Center Lok >2014-04-24 23:36:05,902 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated VM OVFs in Data Center Lok >2014-04-24 23:36:05,902 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to update template OVFs in Data Center Lok >2014-04-24 23:36:05,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully updated templates OVFs in Data Center Lok >2014-04-24 23:36:05,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-24 23:36:05,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-24 23:42:18,379 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6a9b2e0e >2014-04-24 23:42:18,381 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 71eccfcf >2014-04-24 23:42:18,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 71eccfcf >2014-04-24 23:42:18,607 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6a9b2e0e >2014-04-24 23:42:18,612 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 00:36:04,762 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-04-25 00:36:05,903 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VMs/Templates Ovf. >2014-04-25 00:36:05,904 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update VM OVFs in Data Center Lok >2014-04-25 00:36:05,905 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated VM OVFs in Data Center Lok >2014-04-25 00:36:05,905 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to update template OVFs in Data Center Lok >2014-04-25 00:36:05,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully updated templates OVFs in Data Center Lok >2014-04-25 00:36:05,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 00:36:05,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-19) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 00:42:18,616 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 71b53d46 >2014-04-25 00:42:18,617 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 31c9f8fc >2014-04-25 00:42:18,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 31c9f8fc >2014-04-25 00:42:18,842 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 71b53d46 >2014-04-25 00:42:18,847 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 01:36:04,764 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-04-25 01:36:05,906 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VMs/Templates Ovf. >2014-04-25 01:36:05,907 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update VM OVFs in Data Center Lok >2014-04-25 01:36:05,908 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated VM OVFs in Data Center Lok >2014-04-25 01:36:05,908 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to update template OVFs in Data Center Lok >2014-04-25 01:36:05,909 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully updated templates OVFs in Data Center Lok >2014-04-25 01:36:05,909 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 01:36:05,909 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-39) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 01:42:18,850 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 518967d >2014-04-25 01:42:19,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 21d71125 >2014-04-25 01:42:19,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 21d71125 >2014-04-25 01:42:19,323 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 518967d >2014-04-25 01:42:19,328 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 02:36:04,766 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-44) Start refreshing all users data >2014-04-25 02:36:05,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-04-25 02:36:05,910 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center Lok >2014-04-25 02:36:05,911 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center Lok >2014-04-25 02:36:05,911 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center Lok >2014-04-25 02:36:05,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center Lok >2014-04-25 02:36:05,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 02:36:05,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 02:42:19,334 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2494c3f5 >2014-04-25 02:42:19,335 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 22cd1c96 >2014-04-25 02:42:19,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 22cd1c96 >2014-04-25 02:42:19,560 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2494c3f5 >2014-04-25 02:42:19,565 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-30) Start deleteAgedOutAuditLogs >2014-04-25 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-30) Finished deleteAgedOutAuditLogs >2014-04-25 03:36:04,768 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-04-25 03:36:05,912 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-04-25 03:36:05,913 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-04-25 03:36:05,914 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-04-25 03:36:05,914 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-04-25 03:36:05,915 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-04-25 03:36:05,915 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 03:36:05,915 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 03:42:19,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 65d56bae >2014-04-25 03:42:19,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1e90a0bd >2014-04-25 03:42:19,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1e90a0bd >2014-04-25 03:42:19,792 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 65d56bae >2014-04-25 03:42:19,798 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 04:36:04,772 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-41) Start refreshing all users data >2014-04-25 04:36:05,915 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-25 04:36:05,916 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-25 04:36:05,917 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-25 04:36:05,917 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-25 04:36:05,917 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-25 04:36:05,917 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 04:36:05,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 04:42:19,808 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: ebdd958 >2014-04-25 04:42:19,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 520827f7 >2014-04-25 04:42:20,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 520827f7 >2014-04-25 04:42:20,034 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ebdd958 >2014-04-25 04:42:20,039 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 05:36:04,774 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-04-25 05:36:05,916 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-04-25 05:36:05,917 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VM OVFs in Data Center Lok >2014-04-25 05:36:05,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated VM OVFs in Data Center Lok >2014-04-25 05:36:05,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update template OVFs in Data Center Lok >2014-04-25 05:36:05,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated templates OVFs in Data Center Lok >2014-04-25 05:36:05,918 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 05:36:05,919 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 05:42:20,044 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 400d07f >2014-04-25 05:42:20,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 517843e6 >2014-04-25 05:42:20,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 517843e6 >2014-04-25 05:42:20,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 400d07f >2014-04-25 05:42:20,278 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 06:36:04,776 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-99) Start refreshing all users data >2014-04-25 06:36:05,920 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-04-25 06:36:05,922 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VM OVFs in Data Center Lok >2014-04-25 06:36:05,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated VM OVFs in Data Center Lok >2014-04-25 06:36:05,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update template OVFs in Data Center Lok >2014-04-25 06:36:05,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated templates OVFs in Data Center Lok >2014-04-25 06:36:05,923 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 06:36:05,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 06:42:20,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: ceb7745 >2014-04-25 06:42:20,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 36e359a3 >2014-04-25 06:42:20,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 36e359a3 >2014-04-25 06:42:20,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ceb7745 >2014-04-25 06:42:20,515 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 07:36:04,777 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-04-25 07:36:05,924 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-04-25 07:36:05,925 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VM OVFs in Data Center Lok >2014-04-25 07:36:05,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated VM OVFs in Data Center Lok >2014-04-25 07:36:05,926 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update template OVFs in Data Center Lok >2014-04-25 07:36:05,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated templates OVFs in Data Center Lok >2014-04-25 07:36:05,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 07:36:05,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 07:42:20,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 237cdca1 >2014-04-25 07:42:20,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5a78e03a >2014-04-25 07:42:20,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a78e03a >2014-04-25 07:42:20,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 237cdca1 >2014-04-25 07:42:20,754 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 08:36:04,778 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-48) Start refreshing all users data >2014-04-25 08:36:05,927 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-04-25 08:36:05,928 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VM OVFs in Data Center Lok >2014-04-25 08:36:05,929 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated VM OVFs in Data Center Lok >2014-04-25 08:36:05,929 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update template OVFs in Data Center Lok >2014-04-25 08:36:05,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated templates OVFs in Data Center Lok >2014-04-25 08:36:05,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 08:36:05,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 08:42:20,758 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 57533a6f >2014-04-25 08:42:20,759 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 19217a33 >2014-04-25 08:42:20,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 19217a33 >2014-04-25 08:42:20,989 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 57533a6f >2014-04-25 08:42:20,994 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 09:36:04,780 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-04-25 09:36:05,930 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-04-25 09:36:05,932 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VM OVFs in Data Center Lok >2014-04-25 09:36:05,932 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated VM OVFs in Data Center Lok >2014-04-25 09:36:05,932 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update template OVFs in Data Center Lok >2014-04-25 09:36:05,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated templates OVFs in Data Center Lok >2014-04-25 09:36:05,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 09:36:05,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 09:42:20,998 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 378b59a9 >2014-04-25 09:42:20,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1dab82bc >2014-04-25 09:42:21,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1dab82bc >2014-04-25 09:42:21,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 378b59a9 >2014-04-25 09:42:21,239 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 10:36:04,782 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-04-25 10:36:05,933 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-04-25 10:36:05,934 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VM OVFs in Data Center Lok >2014-04-25 10:36:05,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated VM OVFs in Data Center Lok >2014-04-25 10:36:05,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update template OVFs in Data Center Lok >2014-04-25 10:36:05,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated templates OVFs in Data Center Lok >2014-04-25 10:36:05,935 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 10:36:05,936 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 10:42:21,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 62d536dc >2014-04-25 10:42:21,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1c04d835 >2014-04-25 10:42:21,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1c04d835 >2014-04-25 10:42:21,694 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 62d536dc >2014-04-25 10:42:21,699 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 11:36:04,784 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-04-25 11:36:05,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-04-25 11:36:05,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VM OVFs in Data Center Lok >2014-04-25 11:36:05,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated VM OVFs in Data Center Lok >2014-04-25 11:36:05,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update template OVFs in Data Center Lok >2014-04-25 11:36:05,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated templates OVFs in Data Center Lok >2014-04-25 11:36:05,938 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 11:36:05,939 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 11:42:21,702 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2e9529cc >2014-04-25 11:42:21,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 42eb27f8 >2014-04-25 11:42:22,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 42eb27f8 >2014-04-25 11:42:22,069 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e9529cc >2014-04-25 11:42:22,074 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 12:36:04,786 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-25 12:36:05,939 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VMs/Templates Ovf. >2014-04-25 12:36:05,940 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update VM OVFs in Data Center Lok >2014-04-25 12:36:05,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully updated VM OVFs in Data Center Lok >2014-04-25 12:36:05,941 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to update template OVFs in Data Center Lok >2014-04-25 12:36:05,942 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully updated templates OVFs in Data Center Lok >2014-04-25 12:36:05,942 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 12:36:05,942 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-82) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 12:42:22,081 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: c62dd78 >2014-04-25 12:42:22,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2eb9f7a2 >2014-04-25 12:42:22,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2eb9f7a2 >2014-04-25 12:42:22,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: c62dd78 >2014-04-25 12:42:22,405 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 13:36:04,788 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-16) Start refreshing all users data >2014-04-25 13:36:05,942 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VMs/Templates Ovf. >2014-04-25 13:36:05,943 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update VM OVFs in Data Center Lok >2014-04-25 13:36:05,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully updated VM OVFs in Data Center Lok >2014-04-25 13:36:05,944 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to update template OVFs in Data Center Lok >2014-04-25 13:36:05,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully updated templates OVFs in Data Center Lok >2014-04-25 13:36:05,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 13:36:05,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-48) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 13:42:22,410 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: ac93cd4 >2014-04-25 13:42:22,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 21b887d4 >2014-04-25 13:42:22,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 21b887d4 >2014-04-25 13:42:22,853 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ac93cd4 >2014-04-25 13:42:22,858 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 14:36:04,790 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) Start refreshing all users data >2014-04-25 14:36:05,945 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-04-25 14:36:05,946 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VM OVFs in Data Center Lok >2014-04-25 14:36:05,947 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated VM OVFs in Data Center Lok >2014-04-25 14:36:05,947 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update template OVFs in Data Center Lok >2014-04-25 14:36:05,947 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated templates OVFs in Data Center Lok >2014-04-25 14:36:05,947 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 14:36:05,948 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 14:42:22,861 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6c671329 >2014-04-25 14:42:22,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 631b2058 >2014-04-25 14:42:23,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 631b2058 >2014-04-25 14:42:23,092 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6c671329 >2014-04-25 14:42:23,096 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 15:36:04,793 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-69) Start refreshing all users data >2014-04-25 15:36:05,949 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VMs/Templates Ovf. >2014-04-25 15:36:05,949 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update VM OVFs in Data Center Lok >2014-04-25 15:36:05,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated VM OVFs in Data Center Lok >2014-04-25 15:36:05,950 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to update template OVFs in Data Center Lok >2014-04-25 15:36:05,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully updated templates OVFs in Data Center Lok >2014-04-25 15:36:05,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 15:36:05,951 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-40) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 15:42:23,099 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 64132e90 >2014-04-25 15:42:23,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 27562344 >2014-04-25 15:42:23,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 27562344 >2014-04-25 15:42:23,322 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 64132e90 >2014-04-25 15:42:23,326 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 16:36:04,795 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-76) Start refreshing all users data >2014-04-25 16:36:05,952 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VMs/Templates Ovf. >2014-04-25 16:36:05,952 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update VM OVFs in Data Center Lok >2014-04-25 16:36:05,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated VM OVFs in Data Center Lok >2014-04-25 16:36:05,953 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to update template OVFs in Data Center Lok >2014-04-25 16:36:05,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully updated templates OVFs in Data Center Lok >2014-04-25 16:36:05,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 16:36:05,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-78) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 16:42:23,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4cd055e9 >2014-04-25 16:42:23,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1b85b65 >2014-04-25 16:42:23,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b85b65 >2014-04-25 16:42:23,567 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4cd055e9 >2014-04-25 16:42:23,578 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 17:36:04,796 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Start refreshing all users data >2014-04-25 17:36:05,954 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VMs/Templates Ovf. >2014-04-25 17:36:05,955 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update VM OVFs in Data Center Lok >2014-04-25 17:36:05,956 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully updated VM OVFs in Data Center Lok >2014-04-25 17:36:05,956 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to update template OVFs in Data Center Lok >2014-04-25 17:36:05,956 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully updated templates OVFs in Data Center Lok >2014-04-25 17:36:05,957 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 17:36:05,957 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-81) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 17:42:23,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 30d268ef >2014-04-25 17:42:23,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 45e73bf3 >2014-04-25 17:42:23,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 45e73bf3 >2014-04-25 17:42:23,805 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 30d268ef >2014-04-25 17:42:23,811 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 18:36:04,797 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-11) Start refreshing all users data >2014-04-25 18:36:05,957 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-04-25 18:36:05,958 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VM OVFs in Data Center Lok >2014-04-25 18:36:05,959 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated VM OVFs in Data Center Lok >2014-04-25 18:36:05,959 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update template OVFs in Data Center Lok >2014-04-25 18:36:05,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated templates OVFs in Data Center Lok >2014-04-25 18:36:05,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 18:36:05,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 18:42:23,816 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: a5f1c99 >2014-04-25 18:42:23,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6a4e8620 >2014-04-25 18:42:24,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6a4e8620 >2014-04-25 18:42:24,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: a5f1c99 >2014-04-25 18:42:24,278 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 19:36:04,799 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-04-25 19:36:05,960 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-25 19:36:05,962 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-25 19:36:05,962 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-25 19:36:05,962 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-25 19:36:05,963 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-25 19:36:05,963 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 19:36:05,964 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 19:42:24,281 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 77c5d386 >2014-04-25 19:42:24,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5cc23fd8 >2014-04-25 19:42:24,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5cc23fd8 >2014-04-25 19:42:24,511 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 77c5d386 >2014-04-25 19:42:24,515 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 20:36:04,804 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) Start refreshing all users data >2014-04-25 20:36:05,964 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-04-25 20:36:05,965 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VM OVFs in Data Center Lok >2014-04-25 20:36:05,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated VM OVFs in Data Center Lok >2014-04-25 20:36:05,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update template OVFs in Data Center Lok >2014-04-25 20:36:05,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated templates OVFs in Data Center Lok >2014-04-25 20:36:05,966 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 20:36:05,967 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 20:42:24,521 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 68595867 >2014-04-25 20:42:24,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 41bb07b6 >2014-04-25 20:42:24,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 41bb07b6 >2014-04-25 20:42:24,748 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 68595867 >2014-04-25 20:42:24,753 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 21:36:04,808 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-7) Start refreshing all users data >2014-04-25 21:36:05,967 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-04-25 21:36:05,972 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-04-25 21:36:05,972 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-04-25 21:36:05,972 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-04-25 21:36:05,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-04-25 21:36:05,973 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 21:36:05,974 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 21:42:24,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1d1e8696 >2014-04-25 21:42:24,757 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3f15f0c9 >2014-04-25 21:42:24,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3f15f0c9 >2014-04-25 21:42:24,981 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1d1e8696 >2014-04-25 21:42:24,986 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 22:36:04,810 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-04-25 22:36:05,974 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VMs/Templates Ovf. >2014-04-25 22:36:05,975 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update VM OVFs in Data Center Lok >2014-04-25 22:36:05,976 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated VM OVFs in Data Center Lok >2014-04-25 22:36:05,976 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to update template OVFs in Data Center Lok >2014-04-25 22:36:05,977 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully updated templates OVFs in Data Center Lok >2014-04-25 22:36:05,977 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 22:36:05,978 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 22:42:24,992 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1836b03d >2014-04-25 22:42:24,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2e6c8136 >2014-04-25 22:42:25,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e6c8136 >2014-04-25 22:42:25,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1836b03d >2014-04-25 22:42:25,220 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-25 23:36:04,813 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-04-25 23:36:05,979 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-04-25 23:36:05,979 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Lok >2014-04-25 23:36:05,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Lok >2014-04-25 23:36:05,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Lok >2014-04-25 23:36:05,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated templates OVFs in Data Center Lok >2014-04-25 23:36:05,981 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-25 23:36:05,982 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-25 23:42:25,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2d2c61ac >2014-04-25 23:42:25,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3cc07a5f >2014-04-25 23:42:25,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3cc07a5f >2014-04-25 23:42:25,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d2c61ac >2014-04-25 23:42:25,456 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 00:36:04,816 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) Start refreshing all users data >2014-04-26 00:36:05,983 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VMs/Templates Ovf. >2014-04-26 00:36:05,984 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update VM OVFs in Data Center Lok >2014-04-26 00:36:05,984 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated VM OVFs in Data Center Lok >2014-04-26 00:36:05,984 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to update template OVFs in Data Center Lok >2014-04-26 00:36:05,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully updated templates OVFs in Data Center Lok >2014-04-26 00:36:05,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 00:36:05,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-34) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 00:42:25,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5b9ced5d >2014-04-26 00:42:25,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1ba3a3a8 >2014-04-26 00:42:25,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1ba3a3a8 >2014-04-26 00:42:25,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5b9ced5d >2014-04-26 00:42:25,693 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 01:36:04,818 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Start refreshing all users data >2014-04-26 01:36:05,985 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VMs/Templates Ovf. >2014-04-26 01:36:05,986 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update VM OVFs in Data Center Lok >2014-04-26 01:36:05,986 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated VM OVFs in Data Center Lok >2014-04-26 01:36:05,986 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to update template OVFs in Data Center Lok >2014-04-26 01:36:05,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully updated templates OVFs in Data Center Lok >2014-04-26 01:36:05,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 01:36:05,987 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 01:42:25,696 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5a37ee06 >2014-04-26 01:42:25,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: debfa30 >2014-04-26 01:42:25,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: debfa30 >2014-04-26 01:42:25,922 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5a37ee06 >2014-04-26 01:42:25,927 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 02:36:04,820 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) [431634a] Start refreshing all users data >2014-04-26 02:36:05,988 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VMs/Templates Ovf. >2014-04-26 02:36:05,988 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update VM OVFs in Data Center Lok >2014-04-26 02:36:05,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully updated VM OVFs in Data Center Lok >2014-04-26 02:36:05,989 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to update template OVFs in Data Center Lok >2014-04-26 02:36:05,990 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully updated templates OVFs in Data Center Lok >2014-04-26 02:36:05,990 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 02:36:05,990 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-46) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 02:42:25,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 75982339 >2014-04-26 02:42:25,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5730af7f >2014-04-26 02:42:26,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5730af7f >2014-04-26 02:42:26,162 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 75982339 >2014-04-26 02:42:26,167 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-58) Start deleteAgedOutAuditLogs >2014-04-26 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-58) Finished deleteAgedOutAuditLogs >2014-04-26 03:36:04,823 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-45) Start refreshing all users data >2014-04-26 03:36:05,990 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-04-26 03:36:05,991 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-04-26 03:36:05,992 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-04-26 03:36:05,992 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-04-26 03:36:05,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-04-26 03:36:05,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 03:36:05,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 03:42:26,171 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: c568c1a >2014-04-26 03:42:26,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 608054a >2014-04-26 03:42:26,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 608054a >2014-04-26 03:42:26,677 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: c568c1a >2014-04-26 03:42:26,682 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 04:36:04,825 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-04-26 04:36:05,993 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-04-26 04:36:05,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VM OVFs in Data Center Lok >2014-04-26 04:36:05,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated VM OVFs in Data Center Lok >2014-04-26 04:36:05,994 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update template OVFs in Data Center Lok >2014-04-26 04:36:05,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated templates OVFs in Data Center Lok >2014-04-26 04:36:05,995 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 04:36:05,996 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 04:42:26,685 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2029749a >2014-04-26 04:42:26,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1a63af5 >2014-04-26 04:42:26,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1a63af5 >2014-04-26 04:42:26,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2029749a >2014-04-26 04:42:26,915 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 05:36:04,827 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-26 05:36:05,997 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-26 05:36:05,998 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-26 05:36:05,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-26 05:36:05,999 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-26 05:36:06,000 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-26 05:36:06,000 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 05:36:06,000 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 05:42:26,919 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 49dd3b95 >2014-04-26 05:42:26,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5b72d915 >2014-04-26 05:42:27,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5b72d915 >2014-04-26 05:42:27,147 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 49dd3b95 >2014-04-26 05:42:27,157 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 06:36:04,829 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) Start refreshing all users data >2014-04-26 06:36:06,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VMs/Templates Ovf. >2014-04-26 06:36:06,001 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update VM OVFs in Data Center Lok >2014-04-26 06:36:06,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated VM OVFs in Data Center Lok >2014-04-26 06:36:06,002 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to update template OVFs in Data Center Lok >2014-04-26 06:36:06,003 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully updated templates OVFs in Data Center Lok >2014-04-26 06:36:06,003 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 06:36:06,003 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 06:42:27,160 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2e42fd67 >2014-04-26 06:42:27,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 300a8ceb >2014-04-26 06:42:27,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 300a8ceb >2014-04-26 06:42:27,393 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e42fd67 >2014-04-26 06:42:27,400 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 07:36:04,831 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-04-26 07:36:06,004 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-26 07:36:06,005 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-04-26 07:36:06,006 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-04-26 07:36:06,006 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-04-26 07:36:06,006 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-04-26 07:36:06,006 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 07:36:06,007 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 07:42:27,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 67c666f7 >2014-04-26 07:42:27,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1292175b >2014-04-26 07:42:27,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1292175b >2014-04-26 07:42:27,628 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 67c666f7 >2014-04-26 07:42:27,633 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 08:36:04,838 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Start refreshing all users data >2014-04-26 08:36:06,008 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update VMs/Templates Ovf. >2014-04-26 08:36:06,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update VM OVFs in Data Center Lok >2014-04-26 08:36:06,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully updated VM OVFs in Data Center Lok >2014-04-26 08:36:06,009 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update template OVFs in Data Center Lok >2014-04-26 08:36:06,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully updated templates OVFs in Data Center Lok >2014-04-26 08:36:06,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 08:36:06,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 08:42:27,636 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3f36cdea >2014-04-26 08:42:27,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2708de12 >2014-04-26 08:42:27,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2708de12 >2014-04-26 08:42:27,865 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3f36cdea >2014-04-26 08:42:27,870 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 09:36:04,840 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-04-26 09:36:06,010 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VMs/Templates Ovf. >2014-04-26 09:36:06,011 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update VM OVFs in Data Center Lok >2014-04-26 09:36:06,012 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated VM OVFs in Data Center Lok >2014-04-26 09:36:06,012 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to update template OVFs in Data Center Lok >2014-04-26 09:36:06,012 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully updated templates OVFs in Data Center Lok >2014-04-26 09:36:06,012 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 09:36:06,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-51) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 09:42:27,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [431634a] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 48b785be >2014-04-26 09:42:27,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [431634a] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 8ada23e >2014-04-26 09:42:28,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) [431634a] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 8ada23e >2014-04-26 09:42:28,105 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) [431634a] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 48b785be >2014-04-26 09:42:28,114 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) [431634a] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 10:36:04,842 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-26 10:36:06,013 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-04-26 10:36:06,014 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VM OVFs in Data Center Lok >2014-04-26 10:36:06,015 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated VM OVFs in Data Center Lok >2014-04-26 10:36:06,015 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update template OVFs in Data Center Lok >2014-04-26 10:36:06,015 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated templates OVFs in Data Center Lok >2014-04-26 10:36:06,015 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 10:36:06,016 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 10:42:28,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: b684753 >2014-04-26 10:42:28,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6b229ffe >2014-04-26 10:42:28,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6b229ffe >2014-04-26 10:42:28,344 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: b684753 >2014-04-26 10:42:28,349 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 11:36:04,845 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-90) Start refreshing all users data >2014-04-26 11:36:06,017 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-04-26 11:36:06,018 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VM OVFs in Data Center Lok >2014-04-26 11:36:06,019 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated VM OVFs in Data Center Lok >2014-04-26 11:36:06,019 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update template OVFs in Data Center Lok >2014-04-26 11:36:06,020 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated templates OVFs in Data Center Lok >2014-04-26 11:36:06,020 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 11:36:06,020 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 11:42:28,353 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7153b6eb >2014-04-26 11:42:28,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2f0e386d >2014-04-26 11:42:28,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2f0e386d >2014-04-26 11:42:28,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7153b6eb >2014-04-26 11:42:28,586 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 12:36:04,847 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-91) Start refreshing all users data >2014-04-26 12:36:06,021 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-04-26 12:36:06,021 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VM OVFs in Data Center Lok >2014-04-26 12:36:06,022 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully updated VM OVFs in Data Center Lok >2014-04-26 12:36:06,022 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update template OVFs in Data Center Lok >2014-04-26 12:36:06,023 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully updated templates OVFs in Data Center Lok >2014-04-26 12:36:06,023 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 12:36:06,023 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 12:42:28,588 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 703247d >2014-04-26 12:42:28,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5f508e23 >2014-04-26 12:42:28,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5f508e23 >2014-04-26 12:42:28,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 703247d >2014-04-26 12:42:28,821 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 13:36:04,849 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-26 13:36:06,023 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-04-26 13:36:06,024 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-04-26 13:36:06,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-04-26 13:36:06,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-04-26 13:36:06,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-04-26 13:36:06,025 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 13:36:06,026 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 13:42:28,824 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1aa96799 >2014-04-26 13:42:28,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 67d07b7b >2014-04-26 13:42:29,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 67d07b7b >2014-04-26 13:42:29,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1aa96799 >2014-04-26 13:42:29,317 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 14:36:04,851 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-82) Start refreshing all users data >2014-04-26 14:36:06,026 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VMs/Templates Ovf. >2014-04-26 14:36:06,027 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update VM OVFs in Data Center Lok >2014-04-26 14:36:06,028 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated VM OVFs in Data Center Lok >2014-04-26 14:36:06,028 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to update template OVFs in Data Center Lok >2014-04-26 14:36:06,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully updated templates OVFs in Data Center Lok >2014-04-26 14:36:06,029 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 14:36:06,030 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 14:42:29,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7e4ce6b9 >2014-04-26 14:42:29,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6abfbc76 >2014-04-26 14:42:29,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6abfbc76 >2014-04-26 14:42:29,553 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7e4ce6b9 >2014-04-26 14:42:29,558 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 15:36:04,852 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-30) Start refreshing all users data >2014-04-26 15:36:06,030 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-04-26 15:36:06,031 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VM OVFs in Data Center Lok >2014-04-26 15:36:06,032 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated VM OVFs in Data Center Lok >2014-04-26 15:36:06,032 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update template OVFs in Data Center Lok >2014-04-26 15:36:06,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated templates OVFs in Data Center Lok >2014-04-26 15:36:06,033 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 15:36:06,034 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 15:42:29,564 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: fdb33bb >2014-04-26 15:42:29,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 21421f >2014-04-26 15:42:29,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 21421f >2014-04-26 15:42:29,787 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: fdb33bb >2014-04-26 15:42:29,792 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 16:36:04,854 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-4) Start refreshing all users data >2014-04-26 16:36:06,035 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VMs/Templates Ovf. >2014-04-26 16:36:06,035 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update VM OVFs in Data Center Lok >2014-04-26 16:36:06,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated VM OVFs in Data Center Lok >2014-04-26 16:36:06,036 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to update template OVFs in Data Center Lok >2014-04-26 16:36:06,037 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully updated templates OVFs in Data Center Lok >2014-04-26 16:36:06,037 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 16:36:06,037 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-13) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 16:42:29,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4ae118b3 >2014-04-26 16:42:29,796 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1cb62d >2014-04-26 16:42:30,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1cb62d >2014-04-26 16:42:30,021 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4ae118b3 >2014-04-26 16:42:30,027 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 17:36:04,855 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-04-26 17:36:06,038 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-04-26 17:36:06,039 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VM OVFs in Data Center Lok >2014-04-26 17:36:06,040 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated VM OVFs in Data Center Lok >2014-04-26 17:36:06,040 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update template OVFs in Data Center Lok >2014-04-26 17:36:06,040 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated templates OVFs in Data Center Lok >2014-04-26 17:36:06,040 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 17:36:06,041 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 17:42:30,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 14c3024b >2014-04-26 17:42:30,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 418ff3fa >2014-04-26 17:42:30,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 418ff3fa >2014-04-26 17:42:30,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 14c3024b >2014-04-26 17:42:30,293 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 18:36:04,857 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-04-26 18:36:06,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-26 18:36:06,042 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-26 18:36:06,043 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-26 18:36:06,043 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-26 18:36:06,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-26 18:36:06,044 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 18:36:06,045 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 18:42:30,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 53667c06 >2014-04-26 18:42:30,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 782aeb37 >2014-04-26 18:42:30,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 782aeb37 >2014-04-26 18:42:30,984 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 53667c06 >2014-04-26 18:42:30,993 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 19:36:04,859 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-26 19:36:06,045 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-26 19:36:06,046 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-26 19:36:06,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-26 19:36:06,047 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-26 19:36:06,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-26 19:36:06,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 19:36:06,048 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 19:42:30,996 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 1b58b8c5 >2014-04-26 19:42:30,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5dca9b89 >2014-04-26 19:42:31,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5dca9b89 >2014-04-26 19:42:31,224 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b58b8c5 >2014-04-26 19:42:31,229 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 20:36:04,861 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-04-26 20:36:06,050 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VMs/Templates Ovf. >2014-04-26 20:36:06,051 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update VM OVFs in Data Center Lok >2014-04-26 20:36:06,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated VM OVFs in Data Center Lok >2014-04-26 20:36:06,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to update template OVFs in Data Center Lok >2014-04-26 20:36:06,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully updated templates OVFs in Data Center Lok >2014-04-26 20:36:06,052 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 20:36:06,053 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-47) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 20:42:31,234 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6ff7fbe8 >2014-04-26 20:42:31,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1bb75c6 >2014-04-26 20:42:31,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1bb75c6 >2014-04-26 20:42:31,463 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ff7fbe8 >2014-04-26 20:42:31,468 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 21:36:04,863 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-42) Start refreshing all users data >2014-04-26 21:36:06,053 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-04-26 21:36:06,054 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VM OVFs in Data Center Lok >2014-04-26 21:36:06,055 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated VM OVFs in Data Center Lok >2014-04-26 21:36:06,055 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update template OVFs in Data Center Lok >2014-04-26 21:36:06,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated templates OVFs in Data Center Lok >2014-04-26 21:36:06,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 21:36:06,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 21:42:31,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 43eb7fe3 >2014-04-26 21:42:31,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1a1932eb >2014-04-26 21:42:32,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1a1932eb >2014-04-26 21:42:32,011 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 43eb7fe3 >2014-04-26 21:42:32,015 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 22:36:04,865 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-04-26 22:36:06,056 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VMs/Templates Ovf. >2014-04-26 22:36:06,057 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update VM OVFs in Data Center Lok >2014-04-26 22:36:06,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated VM OVFs in Data Center Lok >2014-04-26 22:36:06,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to update template OVFs in Data Center Lok >2014-04-26 22:36:06,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully updated templates OVFs in Data Center Lok >2014-04-26 22:36:06,058 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 22:36:06,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-72) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 22:42:32,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 178d0916 >2014-04-26 22:42:32,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2c4f8f6e >2014-04-26 22:42:32,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2c4f8f6e >2014-04-26 22:42:32,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 178d0916 >2014-04-26 22:42:32,254 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-26 23:36:04,867 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-04-26 23:36:06,059 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VMs/Templates Ovf. >2014-04-26 23:36:06,060 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update VM OVFs in Data Center Lok >2014-04-26 23:36:06,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated VM OVFs in Data Center Lok >2014-04-26 23:36:06,061 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to update template OVFs in Data Center Lok >2014-04-26 23:36:06,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully updated templates OVFs in Data Center Lok >2014-04-26 23:36:06,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-26 23:36:06,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-62) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-26 23:42:32,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7af81ad5 >2014-04-26 23:42:32,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 36a9d505 >2014-04-26 23:42:32,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 36a9d505 >2014-04-26 23:42:32,483 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7af81ad5 >2014-04-26 23:42:32,487 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 00:36:04,869 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) Start refreshing all users data >2014-04-27 00:36:06,062 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-27 00:36:06,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center Lok >2014-04-27 00:36:06,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center Lok >2014-04-27 00:36:06,063 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center Lok >2014-04-27 00:36:06,064 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center Lok >2014-04-27 00:36:06,064 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 00:36:06,064 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 00:42:32,490 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 40ab9fff >2014-04-27 00:42:32,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 43953f84 >2014-04-27 00:42:32,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 43953f84 >2014-04-27 00:42:32,747 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 40ab9fff >2014-04-27 00:42:32,752 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 01:36:04,870 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-14) Start refreshing all users data >2014-04-27 01:36:06,064 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VMs/Templates Ovf. >2014-04-27 01:36:06,065 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update VM OVFs in Data Center Lok >2014-04-27 01:36:06,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully updated VM OVFs in Data Center Lok >2014-04-27 01:36:06,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to update template OVFs in Data Center Lok >2014-04-27 01:36:06,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully updated templates OVFs in Data Center Lok >2014-04-27 01:36:06,066 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 01:36:06,067 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-36) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 01:42:32,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2e574906 >2014-04-27 01:42:32,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 647b40ae >2014-04-27 01:42:32,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 647b40ae >2014-04-27 01:42:32,985 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e574906 >2014-04-27 01:42:32,989 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 02:36:04,872 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) [431634a] Start refreshing all users data >2014-04-27 02:36:06,068 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-04-27 02:36:06,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VM OVFs in Data Center Lok >2014-04-27 02:36:06,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated VM OVFs in Data Center Lok >2014-04-27 02:36:06,069 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update template OVFs in Data Center Lok >2014-04-27 02:36:06,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated templates OVFs in Data Center Lok >2014-04-27 02:36:06,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 02:36:06,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 02:42:32,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7529d911 >2014-04-27 02:42:32,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: e6a272f >2014-04-27 02:42:33,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: e6a272f >2014-04-27 02:42:33,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7529d911 >2014-04-27 02:42:33,224 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 03:35:35,000 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-22) Start deleteAgedOutAuditLogs >2014-04-27 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-22) Finished deleteAgedOutAuditLogs >2014-04-27 03:36:04,875 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-18) Start refreshing all users data >2014-04-27 03:36:06,070 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-04-27 03:36:06,071 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VM OVFs in Data Center Lok >2014-04-27 03:36:06,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated VM OVFs in Data Center Lok >2014-04-27 03:36:06,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update template OVFs in Data Center Lok >2014-04-27 03:36:06,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated templates OVFs in Data Center Lok >2014-04-27 03:36:06,072 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 03:36:06,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 03:42:33,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2db13430 >2014-04-27 03:42:33,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4cd952bf >2014-04-27 03:42:33,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4cd952bf >2014-04-27 03:42:33,453 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [35b64ebe] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2db13430 >2014-04-27 03:42:33,458 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) [35b64ebe] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 04:36:04,877 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-76) Start refreshing all users data >2014-04-27 04:36:06,073 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-04-27 04:36:06,074 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VM OVFs in Data Center Lok >2014-04-27 04:36:06,074 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated VM OVFs in Data Center Lok >2014-04-27 04:36:06,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update template OVFs in Data Center Lok >2014-04-27 04:36:06,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated templates OVFs in Data Center Lok >2014-04-27 04:36:06,075 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 04:36:06,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 04:42:33,461 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 66a7f586 >2014-04-27 04:42:33,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1ce86511 >2014-04-27 04:42:33,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1ce86511 >2014-04-27 04:42:33,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 66a7f586 >2014-04-27 04:42:33,693 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 05:36:04,879 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) Start refreshing all users data >2014-04-27 05:36:06,076 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VMs/Templates Ovf. >2014-04-27 05:36:06,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update VM OVFs in Data Center Lok >2014-04-27 05:36:06,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated VM OVFs in Data Center Lok >2014-04-27 05:36:06,077 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to update template OVFs in Data Center Lok >2014-04-27 05:36:06,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully updated templates OVFs in Data Center Lok >2014-04-27 05:36:06,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 05:36:06,078 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-86) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 05:42:33,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 60bb719d >2014-04-27 05:42:33,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 32202c08 >2014-04-27 05:42:33,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 32202c08 >2014-04-27 05:42:33,923 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 60bb719d >2014-04-27 05:42:33,928 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 06:36:04,881 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-04-27 06:36:06,079 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VMs/Templates Ovf. >2014-04-27 06:36:06,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update VM OVFs in Data Center Lok >2014-04-27 06:36:06,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated VM OVFs in Data Center Lok >2014-04-27 06:36:06,080 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to update template OVFs in Data Center Lok >2014-04-27 06:36:06,081 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully updated templates OVFs in Data Center Lok >2014-04-27 06:36:06,081 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 06:36:06,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 06:42:33,931 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5199293f >2014-04-27 06:42:33,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1e6310f3 >2014-04-27 06:42:34,161 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1e6310f3 >2014-04-27 06:42:34,161 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5199293f >2014-04-27 06:42:34,166 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 07:36:04,884 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-04-27 07:36:06,082 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VMs/Templates Ovf. >2014-04-27 07:36:06,083 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update VM OVFs in Data Center Lok >2014-04-27 07:36:06,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated VM OVFs in Data Center Lok >2014-04-27 07:36:06,084 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to update template OVFs in Data Center Lok >2014-04-27 07:36:06,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully updated templates OVFs in Data Center Lok >2014-04-27 07:36:06,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 07:36:06,085 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-28) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 07:42:34,170 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 34071b83 >2014-04-27 07:42:34,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3e9f5024 >2014-04-27 07:42:34,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3e9f5024 >2014-04-27 07:42:34,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 34071b83 >2014-04-27 07:42:34,403 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 08:36:04,885 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-47) Start refreshing all users data >2014-04-27 08:36:06,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VMs/Templates Ovf. >2014-04-27 08:36:06,086 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update VM OVFs in Data Center Lok >2014-04-27 08:36:06,087 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated VM OVFs in Data Center Lok >2014-04-27 08:36:06,087 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to update template OVFs in Data Center Lok >2014-04-27 08:36:06,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully updated templates OVFs in Data Center Lok >2014-04-27 08:36:06,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 08:36:06,088 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 08:42:34,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 476074ba >2014-04-27 08:42:34,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: d8f83f2 >2014-04-27 08:42:34,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: d8f83f2 >2014-04-27 08:42:34,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 476074ba >2014-04-27 08:42:34,637 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 09:36:04,887 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-27 09:36:06,089 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VMs/Templates Ovf. >2014-04-27 09:36:06,090 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update VM OVFs in Data Center Lok >2014-04-27 09:36:06,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated VM OVFs in Data Center Lok >2014-04-27 09:36:06,091 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to update template OVFs in Data Center Lok >2014-04-27 09:36:06,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully updated templates OVFs in Data Center Lok >2014-04-27 09:36:06,092 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 09:36:06,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-14) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 09:42:34,641 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3b44cc0c >2014-04-27 09:42:34,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 21b0463a >2014-04-27 09:42:34,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 21b0463a >2014-04-27 09:42:34,867 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3b44cc0c >2014-04-27 09:42:34,871 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 10:36:04,889 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-18) Start refreshing all users data >2014-04-27 10:36:06,094 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VMs/Templates Ovf. >2014-04-27 10:36:06,095 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update VM OVFs in Data Center Lok >2014-04-27 10:36:06,096 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated VM OVFs in Data Center Lok >2014-04-27 10:36:06,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to update template OVFs in Data Center Lok >2014-04-27 10:36:06,097 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully updated templates OVFs in Data Center Lok >2014-04-27 10:36:06,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 10:36:06,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-29) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 10:42:34,874 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) [431634a] START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: c0c3e24 >2014-04-27 10:42:34,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) [431634a] START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 77c3b3a9 >2014-04-27 10:42:35,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) [431634a] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 77c3b3a9 >2014-04-27 10:42:35,099 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) [431634a] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: c0c3e24 >2014-04-27 10:42:35,103 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) [431634a] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 11:36:04,891 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) [431634a] Start refreshing all users data >2014-04-27 11:36:06,098 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update VMs/Templates Ovf. >2014-04-27 11:36:06,099 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update VM OVFs in Data Center Lok >2014-04-27 11:36:06,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully updated VM OVFs in Data Center Lok >2014-04-27 11:36:06,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update template OVFs in Data Center Lok >2014-04-27 11:36:06,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully updated templates OVFs in Data Center Lok >2014-04-27 11:36:06,101 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 11:36:06,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 11:42:35,109 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 441f1e91 >2014-04-27 11:42:35,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 570fe487 >2014-04-27 11:42:35,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 570fe487 >2014-04-27 11:42:35,345 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 441f1e91 >2014-04-27 11:42:35,351 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 12:36:04,893 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-41) Start refreshing all users data >2014-04-27 12:36:06,102 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VMs/Templates Ovf. >2014-04-27 12:36:06,103 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update VM OVFs in Data Center Lok >2014-04-27 12:36:06,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated VM OVFs in Data Center Lok >2014-04-27 12:36:06,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to update template OVFs in Data Center Lok >2014-04-27 12:36:06,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully updated templates OVFs in Data Center Lok >2014-04-27 12:36:06,104 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 12:36:06,105 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-43) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 12:42:35,354 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 20006d42 >2014-04-27 12:42:35,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7c051877 >2014-04-27 12:42:35,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c051877 >2014-04-27 12:42:35,577 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 20006d42 >2014-04-27 12:42:35,582 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 13:36:04,895 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-37) Start refreshing all users data >2014-04-27 13:36:06,105 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-27 13:36:06,107 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-27 13:36:06,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-27 13:36:06,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-27 13:36:06,108 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-27 13:36:06,109 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 13:36:06,109 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 13:42:35,586 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 671f10bb >2014-04-27 13:42:35,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 288399e1 >2014-04-27 13:42:35,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 288399e1 >2014-04-27 13:42:35,814 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 671f10bb >2014-04-27 13:42:35,819 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 14:36:04,897 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-54) Start refreshing all users data >2014-04-27 14:36:06,109 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VMs/Templates Ovf. >2014-04-27 14:36:06,110 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update VM OVFs in Data Center Lok >2014-04-27 14:36:06,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully updated VM OVFs in Data Center Lok >2014-04-27 14:36:06,111 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to update template OVFs in Data Center Lok >2014-04-27 14:36:06,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully updated templates OVFs in Data Center Lok >2014-04-27 14:36:06,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 14:36:06,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-31) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 14:42:35,823 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2d7e9547 >2014-04-27 14:42:35,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 517a7fa6 >2014-04-27 14:42:36,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 517a7fa6 >2014-04-27 14:42:36,046 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d7e9547 >2014-04-27 14:42:36,050 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 15:36:04,898 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-58) Start refreshing all users data >2014-04-27 15:36:06,112 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VMs/Templates Ovf. >2014-04-27 15:36:06,113 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update VM OVFs in Data Center Lok >2014-04-27 15:36:06,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated VM OVFs in Data Center Lok >2014-04-27 15:36:06,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to update template OVFs in Data Center Lok >2014-04-27 15:36:06,114 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully updated templates OVFs in Data Center Lok >2014-04-27 15:36:06,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 15:36:06,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-26) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 15:42:36,054 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 148f646a >2014-04-27 15:42:36,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 520975b1 >2014-04-27 15:42:36,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 520975b1 >2014-04-27 15:42:36,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 148f646a >2014-04-27 15:42:36,289 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 16:36:04,899 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-04-27 16:36:06,115 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VMs/Templates Ovf. >2014-04-27 16:36:06,116 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update VM OVFs in Data Center Lok >2014-04-27 16:36:06,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated VM OVFs in Data Center Lok >2014-04-27 16:36:06,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to update template OVFs in Data Center Lok >2014-04-27 16:36:06,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully updated templates OVFs in Data Center Lok >2014-04-27 16:36:06,117 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 16:36:06,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-24) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 16:42:36,293 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5cee649f >2014-04-27 16:42:36,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 27103263 >2014-04-27 16:42:36,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 27103263 >2014-04-27 16:42:36,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5cee649f >2014-04-27 16:42:36,528 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 17:36:04,901 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-29) Start refreshing all users data >2014-04-27 17:36:06,118 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VMs/Templates Ovf. >2014-04-27 17:36:06,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update VM OVFs in Data Center Lok >2014-04-27 17:36:06,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated VM OVFs in Data Center Lok >2014-04-27 17:36:06,120 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to update template OVFs in Data Center Lok >2014-04-27 17:36:06,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully updated templates OVFs in Data Center Lok >2014-04-27 17:36:06,121 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 17:36:06,122 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-44) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 17:42:36,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 3f13d67a >2014-04-27 17:42:36,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2f6603f0 >2014-04-27 17:42:36,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2f6603f0 >2014-04-27 17:42:36,760 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3f13d67a >2014-04-27 17:42:36,765 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 18:36:04,903 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-27) Start refreshing all users data >2014-04-27 18:36:06,123 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VMs/Templates Ovf. >2014-04-27 18:36:06,124 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update VM OVFs in Data Center Lok >2014-04-27 18:36:06,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated VM OVFs in Data Center Lok >2014-04-27 18:36:06,125 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to update template OVFs in Data Center Lok >2014-04-27 18:36:06,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully updated templates OVFs in Data Center Lok >2014-04-27 18:36:06,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 18:36:06,126 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-2) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 18:42:36,769 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 28d50998 >2014-04-27 18:42:36,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3a68ef19 >2014-04-27 18:42:36,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3a68ef19 >2014-04-27 18:42:36,995 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 28d50998 >2014-04-27 18:42:37,001 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 19:36:04,905 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-77) Start refreshing all users data >2014-04-27 19:36:06,127 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VMs/Templates Ovf. >2014-04-27 19:36:06,128 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update VM OVFs in Data Center Lok >2014-04-27 19:36:06,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated VM OVFs in Data Center Lok >2014-04-27 19:36:06,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to update template OVFs in Data Center Lok >2014-04-27 19:36:06,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully updated templates OVFs in Data Center Lok >2014-04-27 19:36:06,129 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 19:36:06,130 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-85) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 19:42:37,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 11c307fb >2014-04-27 19:42:37,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1b40f8d9 >2014-04-27 19:42:37,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1b40f8d9 >2014-04-27 19:42:37,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 11c307fb >2014-04-27 19:42:37,530 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 20:36:04,907 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-81) Start refreshing all users data >2014-04-27 20:36:06,131 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update VMs/Templates Ovf. >2014-04-27 20:36:06,132 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update VM OVFs in Data Center Lok >2014-04-27 20:36:06,132 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully updated VM OVFs in Data Center Lok >2014-04-27 20:36:06,133 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update template OVFs in Data Center Lok >2014-04-27 20:36:06,133 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully updated templates OVFs in Data Center Lok >2014-04-27 20:36:06,133 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 20:36:06,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 20:42:37,535 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 496e946a >2014-04-27 20:42:37,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7ad1e0cf >2014-04-27 20:42:37,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7ad1e0cf >2014-04-27 20:42:37,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 496e946a >2014-04-27 20:42:37,768 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 21:36:04,908 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-15) Start refreshing all users data >2014-04-27 21:36:06,134 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-04-27 21:36:06,135 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VM OVFs in Data Center Lok >2014-04-27 21:36:06,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully updated VM OVFs in Data Center Lok >2014-04-27 21:36:06,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update template OVFs in Data Center Lok >2014-04-27 21:36:06,136 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully updated templates OVFs in Data Center Lok >2014-04-27 21:36:06,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 21:36:06,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 21:42:37,772 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 48e5e792 >2014-04-27 21:42:37,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6dc647a1 >2014-04-27 21:42:38,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6dc647a1 >2014-04-27 21:42:38,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 48e5e792 >2014-04-27 21:42:38,005 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 22:36:04,910 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) Start refreshing all users data >2014-04-27 22:36:06,137 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-04-27 22:36:06,138 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VM OVFs in Data Center Lok >2014-04-27 22:36:06,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated VM OVFs in Data Center Lok >2014-04-27 22:36:06,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update template OVFs in Data Center Lok >2014-04-27 22:36:06,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated templates OVFs in Data Center Lok >2014-04-27 22:36:06,139 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 22:36:06,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 22:42:38,007 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7cafc524 >2014-04-27 22:42:38,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 77b7fe6f >2014-04-27 22:42:38,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 77b7fe6f >2014-04-27 22:42:38,233 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7cafc524 >2014-04-27 22:42:38,238 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-27 23:36:04,912 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-04-27 23:36:06,140 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VMs/Templates Ovf. >2014-04-27 23:36:06,141 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update VM OVFs in Data Center Lok >2014-04-27 23:36:06,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated VM OVFs in Data Center Lok >2014-04-27 23:36:06,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to update template OVFs in Data Center Lok >2014-04-27 23:36:06,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully updated templates OVFs in Data Center Lok >2014-04-27 23:36:06,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-27 23:36:06,142 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-27 23:42:38,242 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6d1d2f11 >2014-04-27 23:42:38,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 72d1ae62 >2014-04-27 23:42:39,041 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 72d1ae62 >2014-04-27 23:42:39,041 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6d1d2f11 >2014-04-27 23:42:39,047 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 00:36:04,914 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-39) Start refreshing all users data >2014-04-28 00:36:06,143 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VMs/Templates Ovf. >2014-04-28 00:36:06,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update VM OVFs in Data Center Lok >2014-04-28 00:36:06,145 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully updated VM OVFs in Data Center Lok >2014-04-28 00:36:06,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to update template OVFs in Data Center Lok >2014-04-28 00:36:06,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully updated templates OVFs in Data Center Lok >2014-04-28 00:36:06,146 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 00:36:06,147 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-95) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 00:42:39,050 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 296b073e >2014-04-28 00:42:39,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 31a88cb0 >2014-04-28 00:42:39,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 31a88cb0 >2014-04-28 00:42:39,278 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 296b073e >2014-04-28 00:42:39,282 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 01:36:04,917 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-24) Start refreshing all users data >2014-04-28 01:36:06,147 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VMs/Templates Ovf. >2014-04-28 01:36:06,148 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update VM OVFs in Data Center Lok >2014-04-28 01:36:06,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated VM OVFs in Data Center Lok >2014-04-28 01:36:06,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to update template OVFs in Data Center Lok >2014-04-28 01:36:06,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully updated templates OVFs in Data Center Lok >2014-04-28 01:36:06,149 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 01:36:06,150 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 01:42:39,287 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 41701ff8 >2014-04-28 01:42:39,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 2842f458 >2014-04-28 01:42:39,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2842f458 >2014-04-28 01:42:39,517 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 41701ff8 >2014-04-28 01:42:39,523 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 02:36:04,918 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-28 02:36:06,150 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VMs/Templates Ovf. >2014-04-28 02:36:06,151 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update VM OVFs in Data Center Lok >2014-04-28 02:36:06,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated VM OVFs in Data Center Lok >2014-04-28 02:36:06,152 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to update template OVFs in Data Center Lok >2014-04-28 02:36:06,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully updated templates OVFs in Data Center Lok >2014-04-28 02:36:06,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 02:36:06,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-49) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 02:42:39,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 2e1ed4fc >2014-04-28 02:42:39,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6a31040b >2014-04-28 02:42:39,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6a31040b >2014-04-28 02:42:39,755 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e1ed4fc >2014-04-28 02:42:39,761 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-43) Start deleteAgedOutAuditLogs >2014-04-28 03:35:35,002 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-43) Finished deleteAgedOutAuditLogs >2014-04-28 03:36:04,920 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-74) Start refreshing all users data >2014-04-28 03:36:06,153 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-28 03:36:06,154 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center Lok >2014-04-28 03:36:06,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center Lok >2014-04-28 03:36:06,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center Lok >2014-04-28 03:36:06,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center Lok >2014-04-28 03:36:06,155 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 03:36:06,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 03:42:39,767 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5fb40918 >2014-04-28 03:42:39,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 24c137be >2014-04-28 03:42:39,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 24c137be >2014-04-28 03:42:39,993 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5fb40918 >2014-04-28 03:42:39,998 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 04:36:04,922 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-19) Start refreshing all users data >2014-04-28 04:36:06,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VMs/Templates Ovf. >2014-04-28 04:36:06,156 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update VM OVFs in Data Center Lok >2014-04-28 04:36:06,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated VM OVFs in Data Center Lok >2014-04-28 04:36:06,157 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to update template OVFs in Data Center Lok >2014-04-28 04:36:06,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully updated templates OVFs in Data Center Lok >2014-04-28 04:36:06,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 04:36:06,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 04:42:40,001 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 7eed52ee >2014-04-28 04:42:40,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: edfc85c >2014-04-28 04:42:40,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: edfc85c >2014-04-28 04:42:40,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7eed52ee >2014-04-28 04:42:40,228 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 05:36:04,924 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-04-28 05:36:06,158 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-04-28 05:36:06,159 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VM OVFs in Data Center Lok >2014-04-28 05:36:06,160 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated VM OVFs in Data Center Lok >2014-04-28 05:36:06,160 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update template OVFs in Data Center Lok >2014-04-28 05:36:06,161 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated templates OVFs in Data Center Lok >2014-04-28 05:36:06,161 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 05:36:06,162 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 05:42:40,232 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5ca4d61c >2014-04-28 05:42:40,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 49e4f7c8 >2014-04-28 05:42:40,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-49) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 49e4f7c8 >2014-04-28 05:42:40,457 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-49) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5ca4d61c >2014-04-28 05:42:40,462 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-49) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 06:36:04,925 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-55) Start refreshing all users data >2014-04-28 06:36:06,163 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VMs/Templates Ovf. >2014-04-28 06:36:06,164 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update VM OVFs in Data Center Lok >2014-04-28 06:36:06,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated VM OVFs in Data Center Lok >2014-04-28 06:36:06,165 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to update template OVFs in Data Center Lok >2014-04-28 06:36:06,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully updated templates OVFs in Data Center Lok >2014-04-28 06:36:06,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 06:36:06,166 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-93) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 06:42:40,466 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 19d2ff5d >2014-04-28 06:42:40,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6b7b3fe9 >2014-04-28 06:42:40,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6b7b3fe9 >2014-04-28 06:42:40,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 19d2ff5d >2014-04-28 06:42:40,700 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 07:36:04,927 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-31) Start refreshing all users data >2014-04-28 07:36:06,167 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VMs/Templates Ovf. >2014-04-28 07:36:06,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update VM OVFs in Data Center Lok >2014-04-28 07:36:06,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated VM OVFs in Data Center Lok >2014-04-28 07:36:06,168 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to update template OVFs in Data Center Lok >2014-04-28 07:36:06,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully updated templates OVFs in Data Center Lok >2014-04-28 07:36:06,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 07:36:06,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-25) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 07:42:40,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 455376f0 >2014-04-28 07:42:40,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 6aa5aa1e >2014-04-28 07:42:40,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6aa5aa1e >2014-04-28 07:42:40,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 455376f0 >2014-04-28 07:42:40,944 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 08:36:04,929 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) Start refreshing all users data >2014-04-28 08:36:06,169 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VMs/Templates Ovf. >2014-04-28 08:36:06,170 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update VM OVFs in Data Center Lok >2014-04-28 08:36:06,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated VM OVFs in Data Center Lok >2014-04-28 08:36:06,171 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to update template OVFs in Data Center Lok >2014-04-28 08:36:06,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully updated templates OVFs in Data Center Lok >2014-04-28 08:36:06,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 08:36:06,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-87) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 08:42:40,947 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 13bedd37 >2014-04-28 08:42:40,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 57ad5001 >2014-04-28 08:42:41,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 57ad5001 >2014-04-28 08:42:41,189 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 13bedd37 >2014-04-28 08:42:41,194 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 09:36:04,931 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-32) Start refreshing all users data >2014-04-28 09:36:06,172 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VMs/Templates Ovf. >2014-04-28 09:36:06,173 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update VM OVFs in Data Center Lok >2014-04-28 09:36:06,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated VM OVFs in Data Center Lok >2014-04-28 09:36:06,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to update template OVFs in Data Center Lok >2014-04-28 09:36:06,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully updated templates OVFs in Data Center Lok >2014-04-28 09:36:06,174 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 09:36:06,175 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-59) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 09:42:41,200 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 254b73c0 >2014-04-28 09:42:41,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 7c9c5560 >2014-04-28 09:42:41,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 7c9c5560 >2014-04-28 09:42:41,441 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 254b73c0 >2014-04-28 09:42:41,445 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 10:36:04,932 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-28 10:36:06,176 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VMs/Templates Ovf. >2014-04-28 10:36:06,177 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update VM OVFs in Data Center Lok >2014-04-28 10:36:06,178 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated VM OVFs in Data Center Lok >2014-04-28 10:36:06,178 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to update template OVFs in Data Center Lok >2014-04-28 10:36:06,179 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully updated templates OVFs in Data Center Lok >2014-04-28 10:36:06,179 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 10:36:06,180 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-65) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 10:42:41,451 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 723a706 >2014-04-28 10:42:41,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 35aa7e93 >2014-04-28 10:42:41,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 35aa7e93 >2014-04-28 10:42:41,705 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 723a706 >2014-04-28 10:42:41,710 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 11:36:04,933 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-10) Start refreshing all users data >2014-04-28 11:36:06,181 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-28 11:36:06,182 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center Lok >2014-04-28 11:36:06,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center Lok >2014-04-28 11:36:06,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center Lok >2014-04-28 11:36:06,183 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center Lok >2014-04-28 11:36:06,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 11:36:06,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 11:42:41,715 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 672b4c99 >2014-04-28 11:42:41,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 4bb205b9 >2014-04-28 11:42:41,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 4bb205b9 >2014-04-28 11:42:41,956 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 672b4c99 >2014-04-28 11:42:41,962 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 12:36:04,934 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-100) Start refreshing all users data >2014-04-28 12:36:06,184 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VMs/Templates Ovf. >2014-04-28 12:36:06,185 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update VM OVFs in Data Center Lok >2014-04-28 12:36:06,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully updated VM OVFs in Data Center Lok >2014-04-28 12:36:06,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to update template OVFs in Data Center Lok >2014-04-28 12:36:06,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully updated templates OVFs in Data Center Lok >2014-04-28 12:36:06,186 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 12:36:06,187 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-27) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 12:42:41,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 5f5f1600 >2014-04-28 12:42:41,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 5b9d8aea >2014-04-28 12:42:42,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5b9d8aea >2014-04-28 12:42:42,219 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5f5f1600 >2014-04-28 12:42:42,224 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 13:26:24,181 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginAdminUserCommand internal: false. >2014-04-28 13:26:24,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-28 13:26:24,258 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginUserCommand internal: false. >2014-04-28 13:26:24,268 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-28 13:36:04,936 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-64) Start refreshing all users data >2014-04-28 13:36:06,187 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-04-28 13:36:06,188 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VM OVFs in Data Center Lok >2014-04-28 13:36:06,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated VM OVFs in Data Center Lok >2014-04-28 13:36:06,189 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update template OVFs in Data Center Lok >2014-04-28 13:36:06,190 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated templates OVFs in Data Center Lok >2014-04-28 13:36:06,190 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to remove unneeded template/vm OVFs in Data Center Lok >2014-04-28 13:36:06,191 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully removed unneeded template/vm OVFs in Data Center Lok >2014-04-28 13:42:42,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 336902a7 >2014-04-28 13:42:42,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 332d3c4f >2014-04-28 13:42:42,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 332d3c4f >2014-04-28 13:42:42,473 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 336902a7 >2014-04-28 13:42:42,478 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 13:43:42,435 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-9) [1d679707] CanDoAction of action AddVds failed. Reasons:VAR__ACTION__ADD,VAR__TYPE__HOST,$server 10.34.1.251,VDS_CANNOT_ADD_MORE_THEN_ONE_HOST_TO_LOCAL_STORAGE >2014-04-28 13:44:03,967 INFO [org.ovirt.engine.core.bll.storage.AddEmptyStoragePoolCommand] (ajp-/127.0.0.1:8702-3) [7b5eeea3] Running command: AddEmptyStoragePoolCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:44:03,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [7b5eeea3] Correlation ID: 7b5eeea3, Call Stack: null, Custom Event ID: -1, Message: Data Center NFS with Type NFS, Compatibility Version 3.3 and Quota Type DISABLED was added by admin@internal >2014-04-28 13:44:22,682 INFO [org.ovirt.engine.core.bll.AddVdsGroupCommand] (ajp-/127.0.0.1:8702-6) [4937fa2f] Running command: AddVdsGroupCommand internal: false. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-28 13:44:22,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [4937fa2f] Correlation ID: 4937fa2f, Call Stack: null, Custom Event ID: -1, Message: Host cluster NFS was added by admin@internal >2014-04-28 13:44:47,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) Command SpmStatusVDS execution failed. Exception: VDSNetworkException: org.apache.commons.httpclient.NoHttpResponseException: The server 10.34.27.209 failed to respond >2014-04-28 13:44:47,718 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-50) [35c89394] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-28 13:44:47,720 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStatsVDSCommand] (DefaultQuartzScheduler_Worker-19) Command GetStatsVDS execution failed. Exception: VDSNetworkException: java.net.SocketException: Connection reset >2014-04-28 13:44:47,729 ERROR [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) vds::refreshVdsStats Failed getVdsStats, vds = 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209, error = VDSNetworkException: java.net.SocketException: Connection reset >2014-04-28 13:44:47,736 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-19) Failed to refresh VDS , vds = 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209, VDS Network Error, continuing. >java.net.SocketException: Connection reset >2014-04-28 13:44:47,772 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [35c89394] Correlation ID: 35c89394, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center Lok. Setting Data Center status to Non Responsive (On host 10.34.27.209, Error: Network error during communication with the Host.). >2014-04-28 13:44:47,780 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-50) [35c89394] IrsBroker::Failed::GetStoragePoolInfoVDS due to: ConnectException: Connection refused >2014-04-28 13:44:50,742 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetCapabilitiesVDSCommand] (DefaultQuartzScheduler_Worker-14) Command GetCapabilitiesVDS execution failed. Exception: VDSNetworkException: java.net.ConnectException: Connection refused >2014-04-28 13:44:50,744 WARN [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-14) Failed to refresh VDS , vds = 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209, VDS Network Error, continuing. >java.net.ConnectException: Connection refused >2014-04-28 13:44:54,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 411be751 >2014-04-28 13:44:54,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) FINISH, GetHardwareInfoVDSCommand, log id: 411be751 >2014-04-28 13:44:54,604 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-28 13:44:54,832 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-56) [564a1084] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-28 13:44:54,836 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-56) [4eed43a2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-28 13:44:54,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [4eed43a2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, storageType = LOCALFS, connectionList = [{ id: 48bb8365-f600-4eba-8ed7-e823810d8790, connection: /mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3963d4e7 >2014-04-28 13:44:58,277 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM: ()]] > >2014-04-28 13:44:58,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-85) HostName = 10.34.27.209 >2014-04-28 13:44:58,278 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-85) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM: () >2014-04-28 13:45:00,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [4eed43a2] FINISH, ConnectStorageServerVDSCommand, return: {48bb8365-f600-4eba-8ed7-e823810d8790=0}, log id: 3963d4e7 >2014-04-28 13:45:00,348 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-56) [4eed43a2] Host 10.34.27.209 storage connection was succeeded >2014-04-28 13:45:00,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [4eed43a2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 20f15d6c >2014-04-28 13:45:00,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [4eed43a2] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 20f15d6c >2014-04-28 13:45:00,668 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-56) [4eed43a2] Host 10.34.27.209 storage connection was succeeded >2014-04-28 13:45:00,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [4eed43a2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: a2179b2 >2014-04-28 13:45:01,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [4eed43a2] FINISH, ConnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: a2179b2 >2014-04-28 13:45:01,001 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-56) [4eed43a2] Host 10.34.27.209 storage connection was succeeded >2014-04-28 13:45:01,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, vds_spm_id = 1, masterDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterVersion = 1), log id: 341a5e97 >2014-04-28 13:45:01,887 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 341a5e97 >2014-04-28 13:45:02,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [4eed43a2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-28 13:45:02,129 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-56) [a3673a7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:45:02,131 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-56) [21697a9d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:45:08,540 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-32) hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool Lok >2014-04-28 13:45:08,777 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-32) SpmStatus on vds 53fd133a-9dc6-11e2-abd9-001a4a223991: Free >2014-04-28 13:45:08,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-32) starting spm on vds 10.34.27.209, storage pool Lok, prevId 1, LVER 15 >2014-04-28 13:45:08,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-32) START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, prevId=1, prevLVER=15, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 430a95ce >2014-04-28 13:45:09,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-32) spmStart polling started: taskId = edc038c8-1dbe-4a7b-a3cb-6c4e610c9eb5 >2014-04-28 13:45:09,666 WARN [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-5) [68ffe22d] Failed to initiate vdsm-id request on host: java.io.IOException: Command returned failure code 1 during SSH session 'root@10.34.1.251' > at org.ovirt.engine.core.utils.ssh.SSHClient.executeCommand(SSHClient.java:508) [utils.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand.getInstalledVdsIdIfExists(AddVdsCommand.java:404) [bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand.canConnect(AddVdsCommand.java:426) [bll.jar:] > at org.ovirt.engine.core.bll.AddVdsCommand.canDoAction(AddVdsCommand.java:342) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:741) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:356) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.RunAction(Backend.java:353) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view10.RunAction(Unknown Source) [common.jar:] > at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.RunAction(GenericApiGWTServiceImpl.java:131) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196) > at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:172) > at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:233) > at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:72) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64) [utils.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499) > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-04-28 13:45:09,680 INFO [org.ovirt.engine.core.bll.AddVdsCommand] (ajp-/127.0.0.1:8702-5) [68ffe22d] Running command: AddVdsCommand internal: false. Entities affected : ID: a13403b3-db7f-4ec4-9b6a-8ec826bd9a15 Type: VdsGroups >2014-04-28 13:45:09,789 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-5) [382041e0] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 56f84250-7f1e-46c8-924f-66c454fd877a Type: VDS >2014-04-28 13:45:09,804 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-5) [382041e0] Lock freed to object EngineLock [exclusiveLocks= key: f03e56af-56f8-4150-9517-271bec2305d9 value: REGISTER_VDS >, sharedLocks= ] >2014-04-28 13:45:09,805 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [382041e0] Cannot get vdsManager for vdsid=56f84250-7f1e-46c8-924f-66c454fd877a >2014-04-28 13:45:09,806 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [382041e0] START, RemoveVdsVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a), log id: 728f532a >2014-04-28 13:45:09,806 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [382041e0] Cannot get vdsManager for vdsid=56f84250-7f1e-46c8-924f-66c454fd877a >2014-04-28 13:45:09,806 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [382041e0] FINISH, RemoveVdsVDSCommand, log id: 728f532a >2014-04-28 13:45:09,807 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [382041e0] Cannot get vdsManager for vdsid=56f84250-7f1e-46c8-924f-66c454fd877a >2014-04-28 13:45:09,808 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [382041e0] START, AddVdsVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a), log id: 63eec732 >2014-04-28 13:45:09,808 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [382041e0] AddVds - entered , starting logic to add VDS 56f84250-7f1e-46c8-924f-66c454fd877a >2014-04-28 13:45:09,809 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [382041e0] AddVds - VDS 56f84250-7f1e-46c8-924f-66c454fd877a was added, will try to add it to the resource manager >2014-04-28 13:45:09,809 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-5) [382041e0] Entered VdsManager constructor >2014-04-28 13:45:09,824 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-5) [382041e0] Initialize vdsBroker (10.34.1.251,54,321) >2014-04-28 13:45:09,825 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-5) [382041e0] VDS 56f84250-7f1e-46c8-924f-66c454fd877a was added to the Resource Manager >2014-04-28 13:45:09,825 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-5) [382041e0] FINISH, AddVdsVDSCommand, log id: 63eec732 >2014-04-28 13:45:09,832 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [382041e0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.1.251. >2014-04-28 13:45:09,873 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-5) [382041e0] Correlation ID: 68ffe22d, Job ID: 5b2f7056-fdd9-449a-b9c5-d6faf3dcd619, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.1.251 was added by admin@internal. >2014-04-28 13:45:09,886 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [22c11f5b] Lock Acquired to object EngineLock [exclusiveLocks= key: 56f84250-7f1e-46c8-924f-66c454fd877a value: VDS >, sharedLocks= ] >2014-04-28 13:45:09,895 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [22c11f5b] Running command: InstallVdsCommand internal: true. Entities affected : ID: 56f84250-7f1e-46c8-924f-66c454fd877a Type: VDS >2014-04-28 13:45:09,895 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [22c11f5b] Before Installation host 56f84250-7f1e-46c8-924f-66c454fd877a, 10.34.1.251 >2014-04-28 13:45:09,936 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [22c11f5b] START, SetVdsStatusVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, status=Installing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4d70143d >2014-04-28 13:45:09,945 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [22c11f5b] FINISH, SetVdsStatusVDSCommand, log id: 4d70143d >2014-04-28 13:45:10,023 INFO [org.ovirt.engine.core.bll.InstallerMessages] (pool-4-thread-49) [22c11f5b] Installation 10.34.1.251: Connected to host 10.34.1.251 with SSH key fingerprint: b2:73:f4:ff:ab:73:10:56:a4:b1:0c:6e:e6:59:20:d2 >2014-04-28 13:45:10,027 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-49) [22c11f5b] Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Connected to host 10.34.1.251 with SSH key fingerprint: b2:73:f4:ff:ab:73:10:56:a4:b1:0c:6e:e6:59:20:d2. >2014-04-28 13:45:12,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-32) spmStart polling ended: taskId = edc038c8-1dbe-4a7b-a3cb-6c4e610c9eb5 task status = finished >2014-04-28 13:45:12,826 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-90) Initializing Host: 10.34.1.251 >2014-04-28 13:45:12,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-32) spmStart polling ended, spm status: SPM >2014-04-28 13:45:12,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-32) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=edc038c8-1dbe-4a7b-a3cb-6c4e610c9eb5), log id: 5419f923 >2014-04-28 13:45:13,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, HSMClearTaskVDSCommand, log id: 5419f923 >2014-04-28 13:45:13,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-32) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@32ee5c26, log id: 430a95ce >2014-04-28 13:45:13,194 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-32) Initialize Irs proxy from vds: 10.34.27.209 >2014-04-28 13:45:13,200 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host 10.34.27.209 (Address: 10.34.27.209). >2014-04-28 13:45:13,213 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-48) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 72351412 >2014-04-28 13:45:13,460 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-48) -- executeIrsBrokerCommand: Attempting on storage pool 19dcb0b3-4c52-4803-a39e-992b1392da62 >2014-04-28 13:45:13,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (pool-4-thread-48) START, HSMGetAllTasksInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 50e94d22 >2014-04-28 13:45:13,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (pool-4-thread-48) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 50e94d22 >2014-04-28 13:45:13,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-48) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 72351412 >2014-04-28 13:45:13,686 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-48) Discovered no tasks on Storage Pool Lok >2014-04-28 13:45:15,500 INFO [org.ovirt.engine.core.bll.VdsDeploy] (pool-4-thread-49) [22c11f5b] Installation of 10.34.1.251. Executing command via SSH umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True < /var/cache/ovirt-engine/ovirt-host-deploy.tar >2014-04-28 13:45:15,516 INFO [org.ovirt.engine.core.utils.archivers.tar.CachedTar] (pool-4-thread-49) Tarball '/var/cache/ovirt-engine/ovirt-host-deploy.tar' is out of date, re-creating >2014-04-28 13:45:15,760 INFO [org.ovirt.engine.core.utils.ssh.SSHDialog] (pool-4-thread-49) SSH execute root@10.34.1.251 'umask 0077; MYTMP="$(mktemp -t ovirt-XXXXXXXXXX)"; trap "chmod -R u+rwX \"${MYTMP}\" > /dev/null 2>&1; rm -fr \"${MYTMP}\" > /dev/null 2>&1" 0; rm -fr "${MYTMP}" && mkdir "${MYTMP}" && tar --warning=no-timestamp -C "${MYTMP}" -x && "${MYTMP}"/setup DIALOG/dialect=str:machine DIALOG/customization=bool:True' >2014-04-28 13:45:17,358 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Initializing >2014-04-28 13:45:17,362 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Initializing. >2014-04-28 13:45:17,379 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Environment setup >2014-04-28 13:45:17,385 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Environment setup. >2014-04-28 13:45:17,443 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Environment packages setup >2014-04-28 13:45:17,447 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Environment packages setup. >2014-04-28 13:45:24,949 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Status: Downloading Packages >2014-04-28 13:45:24,954 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Status: Downloading Packages. >2014-04-28 13:45:24,967 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Download/Verify: dmidecode-2.12-5.el6_5.x86_64 >2014-04-28 13:45:24,983 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Download/Verify: dmidecode-2.12-5.el6_5.x86_64. >2014-04-28 13:45:24,983 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Status: Check Package Signatures >2014-04-28 13:45:24,997 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Status: Check Package Signatures. >2014-04-28 13:45:24,997 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Status: Running Test Transaction >2014-04-28 13:45:25,002 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Status: Running Test Transaction. >2014-04-28 13:45:25,191 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Status: Running Transaction >2014-04-28 13:45:25,197 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Status: Running Transaction. >2014-04-28 13:45:25,764 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum update: 1/2: 1:dmidecode-2.12-5.el6_5.x86_64 >2014-04-28 13:45:25,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum update: 1/2: 1:dmidecode-2.12-5.el6_5.x86_64. >2014-04-28 13:45:27,686 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum updated: 2/2: dmidecode >2014-04-28 13:45:27,732 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum updated: 2/2: dmidecode. >2014-04-28 13:45:32,267 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Verify: 1/2: dmidecode.x86_64 1:2.12-5.el6_5 - u >2014-04-28 13:45:32,271 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Verify: 1/2: dmidecode.x86_64 1:2.12-5.el6_5 - u. >2014-04-28 13:45:32,426 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Verify: 2/2: dmidecode.x86_64 1:2.11-2.el6 - ud >2014-04-28 13:45:32,430 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Verify: 2/2: dmidecode.x86_64 1:2.11-2.el6 - ud. >2014-04-28 13:45:32,906 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Programs detection >2014-04-28 13:45:32,911 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Programs detection. >2014-04-28 13:45:32,949 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Environment customization >2014-04-28 13:45:32,952 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Environment customization. >2014-04-28 13:45:33,027 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Logs at host located at: '/tmp/ovirt-host-deploy-20140428133936.log' >2014-04-28 13:45:33,031 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Logs at host located at: '/tmp/ovirt-host-deploy-20140428133936.log'. >2014-04-28 13:45:33,079 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Host 10.34.1.251 reports unique id 6F126A68-B5C6-DC11-BBDA-C438B039001C >2014-04-28 13:45:33,106 INFO [org.ovirt.engine.core.bll.VdsDeploy] (VdsDeploy) Assigning unique id 6F126A68-B5C6-DC11-BBDA-C438B039001C to Host 10.34.1.251 >2014-04-28 13:45:33,167 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Setup validation >2014-04-28 13:45:33,176 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Setup validation. >2014-04-28 13:45:33,247 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Hardware supports virtualization >2014-04-28 13:45:33,252 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Hardware supports virtualization. >2014-04-28 13:45:33,474 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Transaction setup >2014-04-28 13:45:33,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Transaction setup. >2014-04-28 13:45:33,588 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Misc configuration >2014-04-28 13:45:33,592 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Misc configuration. >2014-04-28 13:45:33,592 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Package installation >2014-04-28 13:45:33,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Package installation. >2014-04-28 13:45:35,699 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Status: Downloading Packages >2014-04-28 13:45:35,703 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Status: Downloading Packages. >2014-04-28 13:45:35,762 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Download/Verify: qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.8.x86_64 >2014-04-28 13:45:35,767 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Download/Verify: qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.8.x86_64. >2014-04-28 13:45:35,847 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Download/Verify: tuned-0.2.19-13.el6.noarch >2014-04-28 13:45:35,851 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Download/Verify: tuned-0.2.19-13.el6.noarch. >2014-04-28 13:45:35,851 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Status: Check Package Signatures >2014-04-28 13:45:35,854 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Status: Check Package Signatures. >2014-04-28 13:45:35,858 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Status: Running Test Transaction >2014-04-28 13:45:35,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Status: Running Test Transaction. >2014-04-28 13:45:36,103 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Status: Running Transaction >2014-04-28 13:45:36,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Status: Running Transaction. >2014-04-28 13:45:36,670 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum install: 1/2: tuned-0.2.19-13.el6.noarch >2014-04-28 13:45:36,675 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum install: 1/2: tuned-0.2.19-13.el6.noarch. >2014-04-28 13:45:38,018 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum install: 2/2: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.8.x86_64 >2014-04-28 13:45:38,037 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum install: 2/2: 2:qemu-kvm-rhev-tools-0.12.1.2-2.415.el6_5.8.x86_64. >2014-04-28 13:45:43,532 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Verify: 1/2: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.8 - u >2014-04-28 13:45:43,542 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Verify: 1/2: qemu-kvm-rhev-tools.x86_64 2:0.12.1.2-2.415.el6_5.8 - u. >2014-04-28 13:45:43,652 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Yum Verify: 2/2: tuned.noarch 0:0.2.19-13.el6 - u >2014-04-28 13:45:43,659 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Yum Verify: 2/2: tuned.noarch 0:0.2.19-13.el6 - u. >2014-04-28 13:45:43,977 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Misc configuration >2014-04-28 13:45:43,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Misc configuration. >2014-04-28 13:45:44,109 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Setting time >2014-04-28 13:45:44,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Setting time. >2014-04-28 13:45:47,154 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Enrolling certificate >2014-04-28 13:45:47,159 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Enrolling certificate. >2014-04-28 13:45:48,464 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Transaction commit >2014-04-28 13:45:48,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Transaction commit. >2014-04-28 13:45:48,599 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Closing up >2014-04-28 13:45:48,604 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Closing up. >2014-04-28 13:45:51,287 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stopping libvirtd >2014-04-28 13:45:51,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stopping libvirtd. >2014-04-28 13:45:52,921 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Starting vdsm >2014-04-28 13:45:52,929 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Starting vdsm. >2014-04-28 13:45:56,920 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Pre-termination >2014-04-28 13:45:56,927 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Pre-termination. >2014-04-28 13:45:56,944 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140428134556-10.34.1.251-22c11f5b.log' >2014-04-28 13:45:56,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Retrieving installation logs to: '/var/log/ovirt-engine/host-deploy/ovirt-20140428134556-10.34.1.251-22c11f5b.log'. >2014-04-28 13:45:57,296 INFO [org.ovirt.engine.core.bll.InstallerMessages] (VdsDeploy) Installation 10.34.1.251: Stage: Termination >2014-04-28 13:45:57,302 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (VdsDeploy) Correlation ID: 22c11f5b, Call Stack: null, Custom Event ID: -1, Message: Installing Host 10.34.1.251. Stage: Termination. >2014-04-28 13:45:57,466 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [22c11f5b] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:45:57,466 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [22c11f5b] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:45:57,969 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [22c11f5b] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:45:57,969 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [22c11f5b] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:45:58,470 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [22c11f5b] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:45:58,471 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [22c11f5b] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:45:59,402 INFO [org.ovirt.engine.core.bll.network.NetworkConfigurator] (pool-4-thread-49) [22c11f5b] Engine managed to communicate with VDSM agent on host 10.34.1.251 >2014-04-28 13:45:59,899 INFO [org.ovirt.engine.core.bll.network.host.SetupNetworksCommand] (pool-4-thread-49) [e6a05b8] Running command: SetupNetworksCommand internal: true. Entities affected : ID: 56f84250-7f1e-46c8-924f-66c454fd877a Type: VDS >2014-04-28 13:45:59,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (pool-4-thread-49) [e6a05b8] START, SetupNetworksVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, force=false, checkConnectivity=true, conectivityTimeout=120, > networks=[rhevm {id=2f2d79b2-1b03-4bdd-a95f-989569c7d9ea, description=Management Network, comment=null, subnet=null, gateway=null, type=null, vlanId=null, stp=false, dataCenterId=f03e56af-56f8-4150-9517-271bec2305d9, mtu=0, vmNetwork=true, cluster=NetworkCluster {id={clusterId=null, networkId=null}, status=OPERATIONAL, display=false, required=true, migration=false}, providedBy=null, label=null}], > bonds=[], > interfaces=[eth1 {id=7dec2ac8-bf5f-452c-b0c7-4f830cae786b, vdsId=56f84250-7f1e-46c8-924f-66c454fd877a, macAddress=00:1c:c4:38:b0:39, networkName=rhevm, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=10.34.1.251, subnet=255.255.255.0, gateway=null, mtu=1500, bridged=false, speed=1000, type=0, networkImplementationDetails=null}, > eth0 {id=e3151226-4254-44cc-9f16-e28e5bcd2d54, vdsId=56f84250-7f1e-46c8-924f-66c454fd877a, macAddress=00:10:18:30:bd:49, networkName=null, vlanId=null, bonded=null, bondName=null, bondOptions=null, bootProtocol=DHCP, address=, subnet=, gateway=null, mtu=1500, bridged=false, speed=0, type=0, networkImplementationDetails=null}], > removedNetworks=[], > removedBonds=[]), log id: 6c0a86fd >2014-04-28 13:45:59,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetupNetworksVDSCommand] (pool-4-thread-49) [e6a05b8] FINISH, SetupNetworksVDSCommand, log id: 6c0a86fd >2014-04-28 13:46:03,386 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:03,386 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:05,389 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:05,389 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:07,392 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:07,392 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:09,395 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:09,395 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:11,396 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:11,396 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:13,397 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:13,397 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:15,400 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:15,401 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:17,406 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:17,406 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:19,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:19,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:21,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:21,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:23,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:23,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:25,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:25,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:27,420 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:27,420 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:29,423 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:29,423 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:31,426 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:31,426 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:33,427 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:33,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:35,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:35,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:37,434 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:37,434 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:39,437 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:39,437 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:41,440 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:41,440 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:43,442 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:43,443 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:45,446 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:45,446 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:47,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:47,449 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:49,452 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:49,452 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:51,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:51,456 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:53,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:53,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:55,461 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:55,462 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.PollVDSCommand] (pool-4-thread-49) [e6a05b8] Command PollVDS execution failed. Exception: RuntimeException: java.util.concurrent.ExecutionException: java.lang.reflect.InvocationTargetException >2014-04-28 13:46:57,442 INFO [org.ovirt.engine.core.bll.network.host.CommitNetworkChangesCommand] (pool-4-thread-49) [707124de] Running command: CommitNetworkChangesCommand internal: true. Entities affected : ID: 56f84250-7f1e-46c8-924f-66c454fd877a Type: VDS >2014-04-28 13:46:57,446 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetSafeNetworkConfigVDSCommand] (pool-4-thread-49) [707124de] START, SetSafeNetworkConfigVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a), log id: 7d4568a6 >2014-04-28 13:46:57,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetSafeNetworkConfigVDSCommand] (pool-4-thread-49) [707124de] FINISH, SetSafeNetworkConfigVDSCommand, log id: 7d4568a6 >2014-04-28 13:46:57,689 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-49) [707124de] Correlation ID: 707124de, Call Stack: null, Custom Event ID: -1, Message: Network changes were saved on host 10.34.1.251 >2014-04-28 13:46:57,690 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [707124de] START, SetVdsStatusVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, status=Initializing, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3a3d6a0 >2014-04-28 13:46:57,692 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-49) [707124de] FINISH, SetVdsStatusVDSCommand, log id: 3a3d6a0 >2014-04-28 13:46:57,692 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [707124de] After Installation host 10.34.1.251, VDS >2014-04-28 13:46:57,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-49) [707124de] Correlation ID: 22c11f5b, Job ID: 5b2f7056-fdd9-449a-b9c5-d6faf3dcd619, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.1.251 installed >2014-04-28 13:46:57,705 INFO [org.ovirt.engine.core.bll.InstallVdsCommand] (pool-4-thread-49) [707124de] Lock freed to object EngineLock [exclusiveLocks= key: 56f84250-7f1e-46c8-924f-66c454fd877a value: VDS >, sharedLocks= ] >2014-04-28 13:46:58,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) START, GetHardwareInfoVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, vds=Host[10.34.1.251]), log id: 7715096e >2014-04-28 13:46:58,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, GetHardwareInfoVDSCommand, log id: 7715096e >2014-04-28 13:46:58,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) START, GetHardwareInfoVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, vds=Host[10.34.1.251]), log id: 187ba303 >2014-04-28 13:46:58,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) FINISH, GetHardwareInfoVDSCommand, log id: 187ba303 >2014-04-28 13:46:59,011 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.1.251. >2014-04-28 13:46:59,238 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-47) [5ffc7361] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-28 13:46:59,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-47) [5ffc7361] START, SetMOMPolicyParametersVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a), log id: 33fb06bf >2014-04-28 13:46:59,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-47) [5ffc7361] FINISH, SetMOMPolicyParametersVDSCommand, log id: 33fb06bf >2014-04-28 13:46:59,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [5ffc7361] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.1.251. >2014-04-28 13:46:59,569 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-47) [2314d850] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 56f84250-7f1e-46c8-924f-66c454fd877a Type: VDS >2014-04-28 13:46:59,571 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-47) [44c9639a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 56f84250-7f1e-46c8-924f-66c454fd877a Type: VDS >2014-04-28 13:47:18,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-3) START, GetVmsInfoVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 4cd3b26d >2014-04-28 13:47:18,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-3) FINISH, GetVmsInfoVDSCommand, log id: 4cd3b26d >2014-04-28 13:47:20,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-3) START, GetVmsInfoVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 203dbfad >2014-04-28 13:47:21,033 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-3) FINISH, GetVmsInfoVDSCommand, log id: 203dbfad >2014-04-28 13:47:35,982 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-7) [2829b7a0] Lock Acquired to object EngineLock [exclusiveLocks= key: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 value: VM >, sharedLocks= ] >2014-04-28 13:47:36,142 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-49) [2829b7a0] Running command: RemoveVmCommand internal: false. Entities affected : ID: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 Type: VM >2014-04-28 13:47:36,153 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-49) [2829b7a0] START, SetVmStatusVDSCommand( vmId = b066ff4b-970f-4305-94a4-1f32f7fa6fe2, status = ImageLocked), log id: 73c81428 >2014-04-28 13:47:36,171 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-49) [2829b7a0] FINISH, SetVmStatusVDSCommand, log id: 73c81428 >2014-04-28 13:47:36,187 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-49) [2829b7a0] Lock freed to object EngineLock [exclusiveLocks= key: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 value: VM >, sharedLocks= ] >2014-04-28 13:47:36,247 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (pool-4-thread-49) [4a602cc3] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 Type: VM >2014-04-28 13:47:36,282 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (pool-4-thread-49) [6521f84a] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-04-28 13:47:36,420 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-49) [6521f84a] START, DeleteImageGroupVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, imageGroupId = 431c414f-83e8-4ad7-a08b-050ee71f9784, postZeros = false, forceDelete = false), log id: 7c5431ee >2014-04-28 13:47:37,076 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-49) [6521f84a] FINISH, DeleteImageGroupVDSCommand, log id: 7c5431ee >2014-04-28 13:47:37,219 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-49) [6521f84a] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 8311d244-bf68-4359-8fb9-8c9937178e14 >2014-04-28 13:47:37,221 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-4-thread-49) [6521f84a] CommandMultiAsyncTasks::AttachTask: Attaching task 729478c9-d1bf-4a70-97cd-03d6ebca5ae1 to command 8311d244-bf68-4359-8fb9-8c9937178e14. >2014-04-28 13:47:37,241 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-49) [6521f84a] Adding task 729478c9-d1bf-4a70-97cd-03d6ebca5ae1 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-04-28 13:47:37,359 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-49) [6521f84a] Correlation ID: 2829b7a0, Job ID: b14920e1-43d5-4ac9-a4fc-c64017391d87, Call Stack: null, Custom Event ID: -1, Message: VM Gal was successfully removed. >2014-04-28 13:47:37,360 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-49) [6521f84a] BaseAsyncTask::StartPollingTask: Starting to poll task 729478c9-d1bf-4a70-97cd-03d6ebca5ae1. >2014-04-28 13:47:39,682 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-90) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-04-28 13:47:39,982 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-8) [50306907] Lock Acquired to object EngineLock [exclusiveLocks= key: e1bf3c57-9833-43bf-bd6c-78cadaa200af value: VM >, sharedLocks= ] >2014-04-28 13:47:40,072 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-50) [50306907] Running command: RemoveVmCommand internal: false. Entities affected : ID: e1bf3c57-9833-43bf-bd6c-78cadaa200af Type: VM >2014-04-28 13:47:40,076 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-50) [50306907] START, SetVmStatusVDSCommand( vmId = e1bf3c57-9833-43bf-bd6c-78cadaa200af, status = ImageLocked), log id: 18130424 >2014-04-28 13:47:40,081 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-50) [50306907] FINISH, SetVmStatusVDSCommand, log id: 18130424 >2014-04-28 13:47:40,083 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-50) [50306907] Lock freed to object EngineLock [exclusiveLocks= key: e1bf3c57-9833-43bf-bd6c-78cadaa200af value: VM >, sharedLocks= ] >2014-04-28 13:47:40,106 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (pool-4-thread-50) [6a4cc5d6] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: e1bf3c57-9833-43bf-bd6c-78cadaa200af Type: VM >2014-04-28 13:47:40,112 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (pool-4-thread-50) [2cb1d776] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-04-28 13:47:40,133 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-90) SPMAsyncTask::PollTask: Polling task 729478c9-d1bf-4a70-97cd-03d6ebca5ae1 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-04-28 13:47:40,137 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-50) [2cb1d776] START, DeleteImageGroupVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, imageGroupId = 5380030f-88ee-49d6-bc7b-5a9b3d7c405a, postZeros = false, forceDelete = false), log id: 15df17cd >2014-04-28 13:47:40,140 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-90) BaseAsyncTask::OnTaskEndSuccess: Task 729478c9-d1bf-4a70-97cd-03d6ebca5ae1 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-04-28 13:47:40,140 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-90) CommandAsyncTask::EndActionIfNecessary: All tasks of command 8311d244-bf68-4359-8fb9-8c9937178e14 has ended -> executing EndAction >2014-04-28 13:47:40,140 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-90) CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: 8311d244-bf68-4359-8fb9-8c9937178e14): calling EndAction . >2014-04-28 13:47:40,141 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-47) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to EndAction RemoveVm, executionIndex: 0 >2014-04-28 13:47:40,167 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-47) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-04-28 13:47:40,168 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-47) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVm completed, handling the result. >2014-04-28 13:47:40,169 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-47) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVm succeeded, clearing tasks. >2014-04-28 13:47:40,175 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 729478c9-d1bf-4a70-97cd-03d6ebca5ae1 >2014-04-28 13:47:40,177 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) START, SPMClearTaskVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, taskId = 729478c9-d1bf-4a70-97cd-03d6ebca5ae1), log id: 3631b4c7 >2014-04-28 13:47:40,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=729478c9-d1bf-4a70-97cd-03d6ebca5ae1), log id: 56418039 >2014-04-28 13:47:40,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) FINISH, HSMClearTaskVDSCommand, log id: 56418039 >2014-04-28 13:47:40,859 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) FINISH, SPMClearTaskVDSCommand, log id: 3631b4c7 >2014-04-28 13:47:40,866 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) BaseAsyncTask::RemoveTaskFromDB: Removed task 729478c9-d1bf-4a70-97cd-03d6ebca5ae1 from DataBase >2014-04-28 13:47:40,867 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-47) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 8311d244-bf68-4359-8fb9-8c9937178e14 >2014-04-28 13:47:43,149 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-50) [2cb1d776] FINISH, DeleteImageGroupVDSCommand, log id: 15df17cd >2014-04-28 13:47:43,161 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) [2cb1d776] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 3db7dda3-40b5-4fbd-a6ee-a7b5507fb1c7 >2014-04-28 13:47:43,161 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-4-thread-50) [2cb1d776] CommandMultiAsyncTasks::AttachTask: Attaching task 4a7cfed0-9c33-46b7-ad51-cfc2ba032898 to command 3db7dda3-40b5-4fbd-a6ee-a7b5507fb1c7. >2014-04-28 13:47:43,173 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-50) [2cb1d776] Adding task 4a7cfed0-9c33-46b7-ad51-cfc2ba032898 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-04-28 13:47:43,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) [2cb1d776] Correlation ID: 50306907, Job ID: ebcfe892-ec67-4a9f-93ec-a09a944a0de8, Call Stack: null, Custom Event ID: -1, Message: VM RHEL-TEST was successfully removed. >2014-04-28 13:47:43,210 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) [2cb1d776] BaseAsyncTask::StartPollingTask: Starting to poll task 4a7cfed0-9c33-46b7-ad51-cfc2ba032898. >2014-04-28 13:47:43,720 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-9) [3164defd] Lock Acquired to object EngineLock [exclusiveLocks= key: ae3b2806-be1d-4846-8dc6-2fa733c073c7 value: VM >, sharedLocks= ] >2014-04-28 13:47:43,761 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-49) [3164defd] Running command: RemoveVmCommand internal: false. Entities affected : ID: ae3b2806-be1d-4846-8dc6-2fa733c073c7 Type: VM >2014-04-28 13:47:43,766 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-49) [3164defd] START, SetVmStatusVDSCommand( vmId = ae3b2806-be1d-4846-8dc6-2fa733c073c7, status = ImageLocked), log id: 1bb6ed81 >2014-04-28 13:47:43,769 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-49) [3164defd] FINISH, SetVmStatusVDSCommand, log id: 1bb6ed81 >2014-04-28 13:47:43,771 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-49) [3164defd] Lock freed to object EngineLock [exclusiveLocks= key: ae3b2806-be1d-4846-8dc6-2fa733c073c7 value: VM >, sharedLocks= ] >2014-04-28 13:47:43,781 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (pool-4-thread-49) [3f1841e] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: ae3b2806-be1d-4846-8dc6-2fa733c073c7 Type: VM >2014-04-28 13:47:43,783 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (pool-4-thread-49) [2cc976ed] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-04-28 13:47:43,800 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-49) [2cc976ed] START, DeleteImageGroupVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, imageGroupId = 6d92b0e9-b312-4e17-82e4-213b6341db2e, postZeros = false, forceDelete = false), log id: 118b3bdd >2014-04-28 13:47:44,665 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-49) [2cc976ed] FINISH, DeleteImageGroupVDSCommand, log id: 118b3bdd >2014-04-28 13:47:44,671 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-49) [2cc976ed] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command d52528fe-d0ab-4dd0-8a7e-b106c4fbd2cd >2014-04-28 13:47:44,671 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-4-thread-49) [2cc976ed] CommandMultiAsyncTasks::AttachTask: Attaching task 3a8b4a54-2491-4a9a-9a11-4d9ef317ca57 to command d52528fe-d0ab-4dd0-8a7e-b106c4fbd2cd. >2014-04-28 13:47:44,682 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-49) [2cc976ed] Adding task 3a8b4a54-2491-4a9a-9a11-4d9ef317ca57 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-04-28 13:47:44,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-49) [2cc976ed] Correlation ID: 3164defd, Job ID: 73f08fe7-a264-4e99-b997-6db58e9618ac, Call Stack: null, Custom Event ID: -1, Message: VM XP was successfully removed. >2014-04-28 13:47:44,698 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-49) [2cc976ed] BaseAsyncTask::StartPollingTask: Starting to poll task 3a8b4a54-2491-4a9a-9a11-4d9ef317ca57. >2014-04-28 13:47:48,133 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (ajp-/127.0.0.1:8702-8) [530cc651] Lock Acquired to object EngineLock [exclusiveLocks= key: 768dae4e-b462-4c5d-b9ce-29c11226891e value: TEMPLATE >, sharedLocks= ] >2014-04-28 13:47:48,167 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (pool-4-thread-49) [530cc651] Running command: RemoveVmTemplateCommand internal: false. Entities affected : ID: 768dae4e-b462-4c5d-b9ce-29c11226891e Type: VmTemplate >2014-04-28 13:47:48,200 INFO [org.ovirt.engine.core.bll.RemoveAllVmTemplateImageTemplatesCommand] (pool-4-thread-49) [530cc651] Running command: RemoveAllVmTemplateImageTemplatesCommand internal: true. Entities affected : ID: 768dae4e-b462-4c5d-b9ce-29c11226891e Type: VmTemplate >2014-04-28 13:47:48,211 INFO [org.ovirt.engine.core.bll.RemoveTemplateSnapshotCommand] (pool-4-thread-49) [1ae9a968] Running command: RemoveTemplateSnapshotCommand internal: true. Entities affected : ID: e4aced3b-4048-4151-9a29-3e917e9ed8fd Type: Storage >2014-04-28 13:47:48,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-49) [1ae9a968] START, DeleteImageGroupVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, imageGroupId = abd35746-8271-442e-9f00-aedab5cc0895, postZeros = false, forceDelete = false), log id: 116abeda >2014-04-28 13:47:48,700 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-49) [1ae9a968] FINISH, DeleteImageGroupVDSCommand, log id: 116abeda >2014-04-28 13:47:48,740 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-49) [1ae9a968] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 17d771d7-64e7-426b-9926-5567bb1d3fd3 >2014-04-28 13:47:48,740 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-4-thread-49) [1ae9a968] CommandMultiAsyncTasks::AttachTask: Attaching task 94813e8c-0aaf-4435-a925-9827308d8c6c to command 17d771d7-64e7-426b-9926-5567bb1d3fd3. >2014-04-28 13:47:48,755 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-49) [1ae9a968] Adding task 94813e8c-0aaf-4435-a925-9827308d8c6c (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-04-28 13:47:48,776 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-49) [1ae9a968] Correlation ID: 530cc651, Job ID: 1add9149-2449-4bf1-98c6-a29e36dcff3e, Call Stack: null, Custom Event ID: -1, Message: Removal of Template xp-temp was initiated by admin@internal. >2014-04-28 13:47:48,776 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-49) [1ae9a968] BaseAsyncTask::StartPollingTask: Starting to poll task 94813e8c-0aaf-4435-a925-9827308d8c6c. >2014-04-28 13:47:50,143 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-79) Polling and updating Async Tasks: 4 tasks, 3 tasks to poll now >2014-04-28 13:47:50,374 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-79) SPMAsyncTask::PollTask: Polling task 3a8b4a54-2491-4a9a-9a11-4d9ef317ca57 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-04-28 13:47:50,378 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-79) BaseAsyncTask::OnTaskEndSuccess: Task 3a8b4a54-2491-4a9a-9a11-4d9ef317ca57 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-04-28 13:47:50,378 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-79) CommandAsyncTask::EndActionIfNecessary: All tasks of command d52528fe-d0ab-4dd0-8a7e-b106c4fbd2cd has ended -> executing EndAction >2014-04-28 13:47:50,378 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-79) CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: d52528fe-d0ab-4dd0-8a7e-b106c4fbd2cd): calling EndAction . >2014-04-28 13:47:50,378 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-79) SPMAsyncTask::PollTask: Polling task 94813e8c-0aaf-4435-a925-9827308d8c6c (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-04-28 13:47:50,379 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-46) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to EndAction RemoveVm, executionIndex: 0 >2014-04-28 13:47:50,383 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-79) BaseAsyncTask::OnTaskEndSuccess: Task 94813e8c-0aaf-4435-a925-9827308d8c6c (Parent Command RemoveVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-04-28 13:47:50,383 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-79) CommandAsyncTask::EndActionIfNecessary: All tasks of command 17d771d7-64e7-426b-9926-5567bb1d3fd3 has ended -> executing EndAction >2014-04-28 13:47:50,384 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-79) CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: 17d771d7-64e7-426b-9926-5567bb1d3fd3): calling EndAction . >2014-04-28 13:47:50,384 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-47) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to EndAction RemoveVmTemplate, executionIndex: 0 >2014-04-28 13:47:50,387 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-79) SPMAsyncTask::PollTask: Polling task 4a7cfed0-9c33-46b7-ad51-cfc2ba032898 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-04-28 13:47:50,394 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (pool-4-thread-47) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmTemplateCommand >2014-04-28 13:47:50,398 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-79) BaseAsyncTask::OnTaskEndSuccess: Task 4a7cfed0-9c33-46b7-ad51-cfc2ba032898 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-04-28 13:47:50,399 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-79) CommandAsyncTask::EndActionIfNecessary: All tasks of command 3db7dda3-40b5-4fbd-a6ee-a7b5507fb1c7 has ended -> executing EndAction >2014-04-28 13:47:50,399 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-79) CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: 3db7dda3-40b5-4fbd-a6ee-a7b5507fb1c7): calling EndAction . >2014-04-28 13:47:50,399 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-49) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to EndAction RemoveVm, executionIndex: 0 >2014-04-28 13:47:50,406 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-46) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-04-28 13:47:50,408 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-46) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVm completed, handling the result. >2014-04-28 13:47:50,408 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-46) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVm succeeded, clearing tasks. >2014-04-28 13:47:50,411 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-49) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-04-28 13:47:50,412 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-49) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVm completed, handling the result. >2014-04-28 13:47:50,412 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-49) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVm succeeded, clearing tasks. >2014-04-28 13:47:50,414 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-49) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 4a7cfed0-9c33-46b7-ad51-cfc2ba032898 >2014-04-28 13:47:50,414 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-49) START, SPMClearTaskVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, taskId = 4a7cfed0-9c33-46b7-ad51-cfc2ba032898), log id: 27b6c768 >2014-04-28 13:47:50,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-49) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=4a7cfed0-9c33-46b7-ad51-cfc2ba032898), log id: 26899b4 >2014-04-28 13:47:50,417 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-46) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 3a8b4a54-2491-4a9a-9a11-4d9ef317ca57 >2014-04-28 13:47:50,417 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-46) START, SPMClearTaskVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, taskId = 3a8b4a54-2491-4a9a-9a11-4d9ef317ca57), log id: 3f5a5f44 >2014-04-28 13:47:50,420 INFO [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (pool-4-thread-47) Lock freed to object EngineLock [exclusiveLocks= key: 768dae4e-b462-4c5d-b9ce-29c11226891e value: TEMPLATE >, sharedLocks= ] >2014-04-28 13:47:50,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-47) Correlation ID: 530cc651, Job ID: 1add9149-2449-4bf1-98c6-a29e36dcff3e, Call Stack: null, Custom Event ID: -1, Message: Removal of Template xp-temp has been completed. >2014-04-28 13:47:50,426 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-47) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVmTemplate completed, handling the result. >2014-04-28 13:47:50,426 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-47) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVmTemplate succeeded, clearing tasks. >2014-04-28 13:47:50,428 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 94813e8c-0aaf-4435-a925-9827308d8c6c >2014-04-28 13:47:50,428 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) START, SPMClearTaskVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, taskId = 94813e8c-0aaf-4435-a925-9827308d8c6c), log id: 3ae8a6dd >2014-04-28 13:47:50,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-49) FINISH, HSMClearTaskVDSCommand, log id: 26899b4 >2014-04-28 13:47:50,640 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-49) FINISH, SPMClearTaskVDSCommand, log id: 27b6c768 >2014-04-28 13:47:50,644 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-49) BaseAsyncTask::RemoveTaskFromDB: Removed task 4a7cfed0-9c33-46b7-ad51-cfc2ba032898 from DataBase >2014-04-28 13:47:50,644 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-49) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 3db7dda3-40b5-4fbd-a6ee-a7b5507fb1c7 >2014-04-28 13:47:51,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=94813e8c-0aaf-4435-a925-9827308d8c6c), log id: 20ab4957 >2014-04-28 13:47:51,342 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-47) FINISH, HSMClearTaskVDSCommand, log id: 20ab4957 >2014-04-28 13:47:51,343 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-47) FINISH, SPMClearTaskVDSCommand, log id: 3ae8a6dd >2014-04-28 13:47:51,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-46) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=3a8b4a54-2491-4a9a-9a11-4d9ef317ca57), log id: 660f4da7 >2014-04-28 13:47:51,345 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-47) BaseAsyncTask::RemoveTaskFromDB: Removed task 94813e8c-0aaf-4435-a925-9827308d8c6c from DataBase >2014-04-28 13:47:51,346 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-47) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 17d771d7-64e7-426b-9926-5567bb1d3fd3 >2014-04-28 13:47:51,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-46) FINISH, HSMClearTaskVDSCommand, log id: 660f4da7 >2014-04-28 13:47:51,570 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-46) FINISH, SPMClearTaskVDSCommand, log id: 3f5a5f44 >2014-04-28 13:47:51,573 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-46) BaseAsyncTask::RemoveTaskFromDB: Removed task 3a8b4a54-2491-4a9a-9a11-4d9ef317ca57 from DataBase >2014-04-28 13:47:51,573 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-46) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity d52528fe-d0ab-4dd0-8a7e-b106c4fbd2cd >2014-04-28 13:48:25,327 ERROR [org.ovirt.engine.core.bll.RemoveVmTemplateCommand] (ajp-/127.0.0.1:8702-3) [5ba88bab] Error during CanDoActionFailure.: java.lang.NullPointerException > at org.ovirt.engine.core.bll.RemoveVmTemplateCommand.getTemplateExclusiveLockMessage(RemoveVmTemplateCommand.java:216) [bll.jar:] > at org.ovirt.engine.core.bll.RemoveVmTemplateCommand.getExclusiveLocks(RemoveVmTemplateCommand.java:211) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.buildLock(CommandBase.java:1826) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.acquireLockInternal(CommandBase.java:1788) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.acquireLock(CommandBase.java:1761) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.internalCanDoAction(CommandBase.java:741) [bll.jar:] > at org.ovirt.engine.core.bll.CommandBase.canDoActionOnly(CommandBase.java:336) [bll.jar:] > at org.ovirt.engine.core.bll.MultipleActionsRunner.Execute(MultipleActionsRunner.java:75) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runMultipleActionsImpl(Backend.java:544) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.runMultipleActionsImpl(Backend.java:559) [bll.jar:] > at org.ovirt.engine.core.bll.Backend.RunMultipleActions(Backend.java:516) [bll.jar:] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext$Invocation.proceed(InterceptorContext.java:374) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.ovirt.engine.core.bll.interceptors.ThreadLocalSessionCleanerInterceptor.injectWebContextToThreadLocal(ThreadLocalSessionCleanerInterceptor.java:13) [bll.jar:] > at sun.reflect.GeneratedMethodAccessor67.invoke(Unknown Source) [:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at org.jboss.as.ee.component.ManagedReferenceLifecycleMethodInterceptorFactory$ManagedReferenceLifecycleMethodInterceptor.processInvocation(ManagedReferenceLifecycleMethodInterceptorFactory.java:123) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) [jboss-as-ejb3.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) [jboss-invocation.jar:1.1.2.Final-redhat-1] > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) [jboss-as-ee.jar:7.3.1.Final-redhat-3] > at org.ovirt.engine.core.common.interfaces.BackendLocal$$$view10.RunMultipleActions(Unknown Source) [common.jar:] > at org.ovirt.engine.ui.frontend.server.gwt.GenericApiGWTServiceImpl.RunMultipleActions(GenericApiGWTServiceImpl.java:112) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_51] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_51] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_51] > at java.lang.reflect.Method.invoke(Method.java:606) [rt.jar:1.7.0_51] > at com.google.gwt.rpc.server.RPC.invokeAndStreamResponse(RPC.java:196) > at com.google.gwt.rpc.server.RpcServlet.processCall(RpcServlet.java:172) > at com.google.gwt.rpc.server.RpcServlet.processPost(RpcServlet.java:233) > at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:754) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] > at javax.servlet.http.HttpServlet.service(HttpServlet.java:847) [jboss-servlet-api_3.0_spec.jar:1.0.2.Final-redhat-1] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:295) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.ui.frontend.server.gwt.GwtCachingFilter.doFilter(GwtCachingFilter.java:132) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.branding.BrandingFilter.doFilter(BrandingFilter.java:72) > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.ovirt.engine.core.utils.servlet.LocaleFilter.doFilter(LocaleFilter.java:64) [utils.jar:] > at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246) > at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214) > at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230) > at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:149) > at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:499) > at org.jboss.as.web.security.SecurityContextAssociationValve.invoke(SecurityContextAssociationValve.java:169) > at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:145) > at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:97) > at org.jboss.web.rewrite.RewriteValve.invoke(RewriteValve.java:466) > at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:102) > at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:336) > at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:488) > at org.apache.coyote.ajp.AjpProtocol$AjpConnectionHandler.process(AjpProtocol.java:420) > at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:920) > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-04-28 13:48:38,839 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-10) [26f51bca] Lock Acquired to object EngineLock [exclusiveLocks= key: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 value: STORAGE >, sharedLocks= ] >2014-04-28 13:48:38,877 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-49) [26f51bca] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 Type: Storage >2014-04-28 13:48:38,888 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-49) [26f51bca] START, DeactivateStorageDomainVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = 6ba7421b-4b80-4aff-ba01-1c08d3ef4619, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 53dd9171 >2014-04-28 13:48:39,127 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-49) [26f51bca] FINISH, DeactivateStorageDomainVDSCommand, log id: 53dd9171 >2014-04-28 13:48:39,127 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-49) [26f51bca] Lock freed to object EngineLock [exclusiveLocks= key: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 value: STORAGE >, sharedLocks= ] >2014-04-28 13:48:39,138 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-4-thread-46) [26f51bca] After deactivate treatment vds: 10.34.27.209,pool Lok >2014-04-28 13:48:39,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-46) [26f51bca] START, RefreshStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, masterStorageDomainId=e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterVersion=1), log id: 3738dcb5 >2014-04-28 13:48:39,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-46) [26f51bca] FINISH, RefreshStoragePoolVDSCommand, log id: 3738dcb5 >2014-04-28 13:48:39,384 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-49) [444399f6] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:48:39,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-49) [444399f6] START, DisconnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6eb256bf >2014-04-28 13:48:41,181 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-8) [3ec9af54] Lock Acquired to object EngineLock [exclusiveLocks= key: c0532ddb-5faf-4ab6-b93f-c490c588cae4 value: STORAGE >, sharedLocks= ] >2014-04-28 13:48:41,218 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-48) [3ec9af54] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: c0532ddb-5faf-4ab6-b93f-c490c588cae4 Type: Storage >2014-04-28 13:48:41,223 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-48) [3ec9af54] START, DeactivateStorageDomainVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 1f7f2f2c >2014-04-28 13:48:41,464 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-48) [3ec9af54] FINISH, DeactivateStorageDomainVDSCommand, log id: 1f7f2f2c >2014-04-28 13:48:41,464 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-48) [3ec9af54] Lock freed to object EngineLock [exclusiveLocks= key: c0532ddb-5faf-4ab6-b93f-c490c588cae4 value: STORAGE >, sharedLocks= ] >2014-04-28 13:48:41,468 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-4-thread-47) [3ec9af54] After deactivate treatment vds: 10.34.27.209,pool Lok >2014-04-28 13:48:41,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [3ec9af54] START, RefreshStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, masterStorageDomainId=e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterVersion=1), log id: 3e547a8 >2014-04-28 13:48:41,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [3ec9af54] FINISH, RefreshStoragePoolVDSCommand, log id: 3e547a8 >2014-04-28 13:48:41,701 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-48) [d18b18a] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:48:41,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-48) [d18b18a] START, DisconnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64c2861e >2014-04-28 13:48:41,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-49) [444399f6] FINISH, DisconnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 6eb256bf >2014-04-28 13:48:41,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-49) [444399f6] Correlation ID: 26f51bca, Job ID: 70891173-6f30-4f1d-87f8-4b2e0318144f, Call Stack: null, Custom Event ID: -1, Message: Storage Domain iso (Data Center Lok) was deactivated by admin@internal >2014-04-28 13:48:44,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-48) [d18b18a] FINISH, DisconnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 64c2861e >2014-04-28 13:48:44,106 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-48) [d18b18a] Correlation ID: 3ec9af54, Job ID: 7d259abf-c614-4c69-8056-826f2f770f92, Call Stack: null, Custom Event ID: -1, Message: Storage Domain gal (Data Center Lok) was deactivated by admin@internal >2014-04-28 13:48:46,069 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-56) [21697a9d] Setting new tasks map. The map contains now 3 tasks >2014-04-28 13:48:48,620 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (ajp-/127.0.0.1:8702-3) MultipleActionsRunner of type RemoveStorageDomain invoked with no actions >2014-04-28 13:48:48,775 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-49) [1b81436a] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected : ID: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 Type: Storage >2014-04-28 13:48:48,776 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-49) [1b81436a] Start detach storage domain >2014-04-28 13:48:48,778 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-49) [1b81436a] Detach storage domain: before connect >2014-04-28 13:48:48,788 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-48) [60e7b4f8] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:48:48,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [60e7b4f8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3ee359c1 >2014-04-28 13:48:49,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [60e7b4f8] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 3ee359c1 >2014-04-28 13:48:49,090 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-49) [1b81436a] Detach storage domain: after connect >2014-04-28 13:48:49,093 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-4-thread-49) [1b81436a] START, DetachStorageDomainVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = 6ba7421b-4b80-4aff-ba01-1c08d3ef4619, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 2a5e87fa >2014-04-28 13:48:52,256 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (ajp-/127.0.0.1:8702-6) MultipleActionsRunner of type RemoveStorageDomain invoked with no actions >2014-04-28 13:48:52,354 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [71b71680] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected : ID: c0532ddb-5faf-4ab6-b93f-c490c588cae4 Type: Storage >2014-04-28 13:48:52,354 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [71b71680] Start detach storage domain >2014-04-28 13:48:52,357 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [71b71680] Detach storage domain: before connect >2014-04-28 13:48:52,360 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-47) [6e110d1a] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:48:52,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [6e110d1a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 377e1555 >2014-04-28 13:48:52,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [6e110d1a] FINISH, ConnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 377e1555 >2014-04-28 13:48:52,648 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [71b71680] Detach storage domain: after connect >2014-04-28 13:48:52,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-4-thread-46) [71b71680] START, DetachStorageDomainVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 117dc25a >2014-04-28 13:49:12,825 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-4-thread-49) [1b81436a] FINISH, DetachStorageDomainVDSCommand, log id: 2a5e87fa >2014-04-28 13:49:12,825 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-49) [1b81436a] Detach storage domain: after detach in vds >2014-04-28 13:49:12,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-50) [1b81436a] START, RefreshStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, masterStorageDomainId=e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterVersion=1), log id: 5ded1dec >2014-04-28 13:49:13,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-50) [1b81436a] FINISH, RefreshStoragePoolVDSCommand, log id: 5ded1dec >2014-04-28 13:49:13,063 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-50) [6addb94e] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:49:13,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-50) [6addb94e] START, DisconnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2bf9b17f >2014-04-28 13:49:16,069 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-29) Setting new tasks map. The map contains now 0 tasks >2014-04-28 13:49:16,069 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-29) Cleared all tasks of pool 19dcb0b3-4c52-4803-a39e-992b1392da62. >2014-04-28 13:49:16,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-50) [6addb94e] FINISH, DisconnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 2bf9b17f >2014-04-28 13:49:16,548 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-49) [1b81436a] Detach storage domain: after disconnect storage >2014-04-28 13:49:16,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResetISOPathVDSCommand] (pool-4-thread-49) [1b81436a] START, ResetISOPathVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 138c5b79 >2014-04-28 13:49:16,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResetISOPathVDSCommand] (pool-4-thread-49) [1b81436a] FINISH, ResetISOPathVDSCommand, log id: 138c5b79 >2014-04-28 13:49:16,557 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-49) [1b81436a] End detach storage domain >2014-04-28 13:49:16,562 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-49) [1b81436a] Correlation ID: 1b81436a, Job ID: e9e9b918-ee14-42f0-b3a0-3498779a199d, Call Stack: null, Custom Event ID: -1, Message: Storage Domain iso was detached from Data Center Lok by admin@internal >2014-04-28 13:49:18,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-4-thread-46) [71b71680] FINISH, DetachStorageDomainVDSCommand, log id: 117dc25a >2014-04-28 13:49:18,733 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [71b71680] Detach storage domain: after detach in vds >2014-04-28 13:49:18,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-48) [71b71680] START, RefreshStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, masterStorageDomainId=e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterVersion=1), log id: 6cccc59 >2014-04-28 13:49:19,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-48) [71b71680] FINISH, RefreshStoragePoolVDSCommand, log id: 6cccc59 >2014-04-28 13:49:19,182 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-48) [4d64b86f] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:49:19,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-48) [4d64b86f] START, DisconnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5492e594 >2014-04-28 13:49:22,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-48) [4d64b86f] FINISH, DisconnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 5492e594 >2014-04-28 13:49:22,563 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [71b71680] Detach storage domain: after disconnect storage >2014-04-28 13:49:22,568 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [71b71680] End detach storage domain >2014-04-28 13:49:22,572 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [71b71680] Correlation ID: 71b71680, Job ID: 1aad2401-04e4-4eba-82fe-770c861fac76, Call Stack: null, Custom Event ID: -1, Message: Storage Domain gal was detached from Data Center Lok by admin@internal >2014-04-28 13:49:40,998 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [11d398dd] Lock Acquired to object EngineLock [exclusiveLocks= key: e4aced3b-4048-4151-9a29-3e917e9ed8fd value: STORAGE >key: 19dcb0b3-4c52-4803-a39e-992b1392da62 value: POOL >, sharedLocks= ] >2014-04-28 13:49:41,066 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-47) [11d398dd] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: e4aced3b-4048-4151-9a29-3e917e9ed8fd Type: Storage >2014-04-28 13:49:41,079 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-47) [11d398dd] START, DeactivateStorageDomainVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 313e964 >2014-04-28 13:49:41,312 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-47) [11d398dd] FINISH, DeactivateStorageDomainVDSCommand, log id: 313e964 >2014-04-28 13:49:41,313 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-47) [11d398dd] Lock freed to object EngineLock [exclusiveLocks= key: e4aced3b-4048-4151-9a29-3e917e9ed8fd value: STORAGE >key: 19dcb0b3-4c52-4803-a39e-992b1392da62 value: POOL >, sharedLocks= ] >2014-04-28 13:49:41,314 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-4-thread-48) [11d398dd] After deactivate treatment vds: 10.34.27.209,pool Lok >2014-04-28 13:49:41,315 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (pool-4-thread-47) [11d398dd] START, SpmStopOnIrsVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 4687d3a9 >2014-04-28 13:49:41,316 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (pool-4-thread-47) [11d398dd] START, ResetIrsVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, vdsId = 53fd133a-9dc6-11e2-abd9-001a4a223991, ignoreStopFailed = false), log id: 2ae476cc >2014-04-28 13:49:41,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-47) [11d398dd] START, SpmStopVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 1ffb5da >2014-04-28 13:49:41,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-47) [11d398dd] SpmStopVDSCommand::Stopping SPM on vds 10.34.27.209, pool id 19dcb0b3-4c52-4803-a39e-992b1392da62 >2014-04-28 13:49:41,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-47) [11d398dd] FINISH, SpmStopVDSCommand, log id: 1ffb5da >2014-04-28 13:49:41,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (pool-4-thread-47) [11d398dd] FINISH, ResetIrsVDSCommand, log id: 2ae476cc >2014-04-28 13:49:41,793 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (pool-4-thread-47) [11d398dd] FINISH, SpmStopOnIrsVDSCommand, log id: 4687d3a9 >2014-04-28 13:49:41,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (pool-4-thread-47) [11d398dd] START, DisconnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, vds_spm_id = 1), log id: 1448cb7a >2014-04-28 13:49:42,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (pool-4-thread-47) [11d398dd] FINISH, DisconnectStoragePoolVDSCommand, log id: 1448cb7a >2014-04-28 13:49:42,037 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-47) [549fa5f4] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:49:42,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-47) [549fa5f4] START, DisconnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 48bb8365-f600-4eba-8ed7-e823810d8790, connection: /mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5afe1953 >2014-04-28 13:49:44,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-47) [549fa5f4] FINISH, DisconnectStorageServerVDSCommand, return: {48bb8365-f600-4eba-8ed7-e823810d8790=0}, log id: 5afe1953 >2014-04-28 13:49:44,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-47) [549fa5f4] Correlation ID: 11d398dd, Job ID: 6cb6cf4e-fe72-4a53-9cb0-0e12042885ca, Call Stack: null, Custom Event ID: -1, Message: Storage Domain storage (Data Center Lok) was deactivated by admin@internal >2014-04-28 13:50:06,168 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [2e6b4ab9] Lock Acquired to object EngineLock [exclusiveLocks= key: e4aced3b-4048-4151-9a29-3e917e9ed8fd value: STORAGE >, sharedLocks= ] >2014-04-28 13:50:06,185 WARN [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [2e6b4ab9] CanDoAction of action RemoveStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__REMOVE,ERROR_CANNOT_DETACH_STORAGE_DOMAIN_WITH_IMAGES >2014-04-28 13:50:06,185 INFO [org.ovirt.engine.core.bll.storage.RemoveStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [2e6b4ab9] Lock freed to object EngineLock [exclusiveLocks= key: e4aced3b-4048-4151-9a29-3e917e9ed8fd value: STORAGE >, sharedLocks= ] >2014-04-28 13:50:06,200 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (ajp-/127.0.0.1:8702-7) MultipleActionsRunner of type DetachStorageDomainFromPool invoked with no actions >2014-04-28 13:50:10,476 WARN [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (ajp-/127.0.0.1:8702-8) [74735178] CanDoAction of action RemoveStoragePool failed. Reasons:VAR__TYPE__STORAGE__POOL,VAR__ACTION__REMOVE,ERROR_CANNOT_REMOVE_STORAGE_POOL_WITH_IMAGES >2014-04-28 13:50:24,526 WARN [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (ajp-/127.0.0.1:8702-3) [7cbf3e91] CanDoAction of action RemoveStoragePool failed. Reasons:VAR__TYPE__STORAGE__POOL,VAR__ACTION__REMOVE,ERROR_CANNOT_REMOVE_STORAGE_POOL_WITH_IMAGES >2014-04-28 13:50:34,912 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-10) [6356eb3] Lock Acquired to object EngineLock [exclusiveLocks= key: a3fc1295-07f4-45e8-95de-6aef463c0938 value: DISK >, sharedLocks= ] >2014-04-28 13:50:34,915 WARN [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-10) [6356eb3] CanDoAction of action RemoveDisk failed. Reasons:VAR__ACTION__REMOVE,VAR__TYPE__VM_DISK,ACTION_TYPE_FAILED_STORAGE_DOMAIN_STATUS_ILLEGAL >2014-04-28 13:50:34,915 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-10) [6356eb3] Lock freed to object EngineLock [exclusiveLocks= key: a3fc1295-07f4-45e8-95de-6aef463c0938 value: DISK >, sharedLocks= ] >2014-04-28 13:50:46,533 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [77aaa8c1] Lock Acquired to object EngineLock [exclusiveLocks= key: e4aced3b-4048-4151-9a29-3e917e9ed8fd value: STORAGE >, sharedLocks= ] >2014-04-28 13:50:46,552 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [77aaa8c1] Running command: ActivateStorageDomainCommand internal: false. Entities affected : ID: e4aced3b-4048-4151-9a29-3e917e9ed8fd Type: Storage >2014-04-28 13:50:46,556 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [77aaa8c1] Lock freed to object EngineLock [exclusiveLocks= key: e4aced3b-4048-4151-9a29-3e917e9ed8fd value: STORAGE >, sharedLocks= ] >2014-04-28 13:50:46,557 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [77aaa8c1] ActivateStorage Domain. Before Connect all hosts to pool. Time:4/28/14 1:50 PM >2014-04-28 13:50:46,563 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-47) [403f3310] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:50:46,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [403f3310] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 48bb8365-f600-4eba-8ed7-e823810d8790, connection: /mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 75e2e58d >2014-04-28 13:50:46,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [403f3310] FINISH, ConnectStorageServerVDSCommand, return: {48bb8365-f600-4eba-8ed7-e823810d8790=0}, log id: 75e2e58d >2014-04-28 13:50:46,832 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-46) [77aaa8c1] START, ActivateStorageDomainVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd), log id: 345df84d >2014-04-28 13:50:46,839 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (pool-4-thread-46) [456b5a29] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-28 13:50:46,843 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [456b5a29] Correlation ID: 456b5a29, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-04-28 13:50:47,071 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-46) [456b5a29] hostFromVds::selectedVds - 10.34.27.209, spmStatus Unknown_Pool, storage pool Lok >2014-04-28 13:50:47,074 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) [456b5a29] START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, vds_spm_id = 1, masterDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterVersion = 1), log id: 221393df >2014-04-28 13:50:49,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) [456b5a29] FINISH, ConnectStoragePoolVDSCommand, log id: 221393df >2014-04-28 13:50:49,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-46) [456b5a29] hostFromVds::Connected host to pool - selectedVds - 10.34.27.209, spmStatus Free, storage pool Lok >2014-04-28 13:50:49,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-46) [456b5a29] starting spm on vds 10.34.27.209, storage pool Lok, prevId -1, LVER 16 >2014-04-28 13:50:49,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-46) [456b5a29] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, prevId=-1, prevLVER=16, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 4d047c1a >2014-04-28 13:50:50,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-46) [456b5a29] spmStart polling started: taskId = 4054bce8-6fdd-4731-bce4-b641e52cba5f >2014-04-28 13:50:51,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-46) [456b5a29] spmStart polling ended: taskId = 4054bce8-6fdd-4731-bce4-b641e52cba5f task status = finished >2014-04-28 13:50:51,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-46) [456b5a29] spmStart polling ended, spm status: SPM >2014-04-28 13:50:51,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-46) [456b5a29] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=4054bce8-6fdd-4731-bce4-b641e52cba5f), log id: a9a0c82 >2014-04-28 13:50:51,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-46) [456b5a29] FINISH, HSMClearTaskVDSCommand, log id: a9a0c82 >2014-04-28 13:50:51,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-46) [456b5a29] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@287520e9, log id: 4d047c1a >2014-04-28 13:50:51,904 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-46) [456b5a29] Initialize Irs proxy from vds: 10.34.27.209 >2014-04-28 13:50:51,920 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [456b5a29] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host 10.34.27.209 (Address: 10.34.27.209). >2014-04-28 13:50:52,167 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-46) [456b5a29] FINISH, ActivateStorageDomainVDSCommand, log id: 345df84d >2014-04-28 13:50:52,167 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [456b5a29] ActivateStorage Domain. After Connect all hosts to pool. Time:4/28/14 1:50 PM >2014-04-28 13:50:52,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) [456b5a29] START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, vds_spm_id = 1, masterDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterVersion = 1), log id: 2ad13675 >2014-04-28 13:50:52,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) [456b5a29] FINISH, ConnectStoragePoolVDSCommand, log id: 2ad13675 >2014-04-28 13:50:52,426 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [456b5a29] ActivateStorage Domain. After change storage pool status in vds. Time:4/28/14 1:50 PM >2014-04-28 13:50:52,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [456b5a29] Correlation ID: 77aaa8c1, Job ID: d8f4cc8f-a44e-4010-8937-3bb487401446, Call Stack: null, Custom Event ID: -1, Message: Storage Domain storage (Data Center Lok) was activated by admin@internal >2014-04-28 13:51:00,418 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (ajp-/127.0.0.1:8702-4) [454811c8] Lock Acquired to object EngineLock [exclusiveLocks= key: a3fc1295-07f4-45e8-95de-6aef463c0938 value: DISK >, sharedLocks= ] >2014-04-28 13:51:00,439 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (pool-4-thread-50) [454811c8] Running command: RemoveDiskCommand internal: false. Entities affected : ID: a3fc1295-07f4-45e8-95de-6aef463c0938 Type: Disk >2014-04-28 13:51:00,447 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (pool-4-thread-50) [3a7d4f3d] Running command: RemoveImageCommand internal: true. Entities affected : ID: e4aced3b-4048-4151-9a29-3e917e9ed8fd Type: Storage >2014-04-28 13:51:00,479 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-50) [3a7d4f3d] START, DeleteImageGroupVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, imageGroupId = a3fc1295-07f4-45e8-95de-6aef463c0938, postZeros = false, forceDelete = false), log id: 2114c4b6 >2014-04-28 13:51:00,773 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-50) [3a7d4f3d] FINISH, DeleteImageGroupVDSCommand, log id: 2114c4b6 >2014-04-28 13:51:00,785 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) [3a7d4f3d] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 2923a1f5-eb97-41da-8db3-33556dc9ec82 >2014-04-28 13:51:00,785 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-4-thread-50) [3a7d4f3d] CommandMultiAsyncTasks::AttachTask: Attaching task da44279d-90d4-4c60-b74c-467d5f90ded4 to command 2923a1f5-eb97-41da-8db3-33556dc9ec82. >2014-04-28 13:51:00,799 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-50) [3a7d4f3d] Adding task da44279d-90d4-4c60-b74c-467d5f90ded4 (Parent Command RemoveDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-04-28 13:51:00,887 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) [3a7d4f3d] Correlation ID: 454811c8, Job ID: b6c5bdf7-3462-4c36-906b-6d3b1ba6e2ae, Call Stack: null, Custom Event ID: -1, Message: Disk XP_Disk1 was successfully removed from domain storage (User admin@internal. >2014-04-28 13:51:00,887 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) [3a7d4f3d] BaseAsyncTask::StartPollingTask: Starting to poll task da44279d-90d4-4c60-b74c-467d5f90ded4. >2014-04-28 13:51:00,887 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (pool-4-thread-50) [3a7d4f3d] Lock freed to object EngineLock [exclusiveLocks= key: a3fc1295-07f4-45e8-95de-6aef463c0938 value: DISK >, sharedLocks= ] >2014-04-28 13:51:10,405 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-82) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-04-28 13:51:10,654 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-82) SPMAsyncTask::PollTask: Polling task da44279d-90d4-4c60-b74c-467d5f90ded4 (Parent Command RemoveDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-04-28 13:51:10,658 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-82) BaseAsyncTask::OnTaskEndSuccess: Task da44279d-90d4-4c60-b74c-467d5f90ded4 (Parent Command RemoveDisk, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-04-28 13:51:10,658 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-82) CommandAsyncTask::EndActionIfNecessary: All tasks of command 2923a1f5-eb97-41da-8db3-33556dc9ec82 has ended -> executing EndAction >2014-04-28 13:51:10,660 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-82) CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: 2923a1f5-eb97-41da-8db3-33556dc9ec82): calling EndAction . >2014-04-28 13:51:10,660 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to EndAction RemoveDisk, executionIndex: 0 >2014-04-28 13:51:10,666 INFO [org.ovirt.engine.core.bll.RemoveDiskCommand] (pool-4-thread-50) Ending command successfully: org.ovirt.engine.core.bll.RemoveDiskCommand >2014-04-28 13:51:10,666 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveDisk completed, handling the result. >2014-04-28 13:51:10,666 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveDisk succeeded, clearing tasks. >2014-04-28 13:51:10,670 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) SPMAsyncTask::ClearAsyncTask: Attempting to clear task da44279d-90d4-4c60-b74c-467d5f90ded4 >2014-04-28 13:51:10,670 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) START, SPMClearTaskVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, taskId = da44279d-90d4-4c60-b74c-467d5f90ded4), log id: 6cd8bd42 >2014-04-28 13:51:10,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=da44279d-90d4-4c60-b74c-467d5f90ded4), log id: 9716fe2 >2014-04-28 13:51:10,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) FINISH, HSMClearTaskVDSCommand, log id: 9716fe2 >2014-04-28 13:51:10,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) FINISH, SPMClearTaskVDSCommand, log id: 6cd8bd42 >2014-04-28 13:51:10,897 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) BaseAsyncTask::RemoveTaskFromDB: Removed task da44279d-90d4-4c60-b74c-467d5f90ded4 from DataBase >2014-04-28 13:51:10,897 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 2923a1f5-eb97-41da-8db3-33556dc9ec82 >2014-04-28 13:51:15,532 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-5) [14d9fb88] Lock Acquired to object EngineLock [exclusiveLocks= key: e4aced3b-4048-4151-9a29-3e917e9ed8fd value: STORAGE >key: 19dcb0b3-4c52-4803-a39e-992b1392da62 value: POOL >, sharedLocks= ] >2014-04-28 13:51:15,551 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-46) [14d9fb88] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: e4aced3b-4048-4151-9a29-3e917e9ed8fd Type: Storage >2014-04-28 13:51:15,562 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-46) [14d9fb88] START, DeactivateStorageDomainVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, storageDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 680e26ee >2014-04-28 13:51:15,955 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-46) [14d9fb88] FINISH, DeactivateStorageDomainVDSCommand, log id: 680e26ee >2014-04-28 13:51:15,955 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-46) [14d9fb88] Lock freed to object EngineLock [exclusiveLocks= key: e4aced3b-4048-4151-9a29-3e917e9ed8fd value: STORAGE >key: 19dcb0b3-4c52-4803-a39e-992b1392da62 value: POOL >, sharedLocks= ] >2014-04-28 13:51:15,956 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-4-thread-50) [14d9fb88] After deactivate treatment vds: 10.34.27.209,pool Lok >2014-04-28 13:51:15,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (pool-4-thread-46) [14d9fb88] START, SpmStopOnIrsVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 6d6767c1 >2014-04-28 13:51:15,957 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (pool-4-thread-46) [14d9fb88] START, ResetIrsVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false, vdsId = 53fd133a-9dc6-11e2-abd9-001a4a223991, ignoreStopFailed = false), log id: 45946eeb >2014-04-28 13:51:15,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-46) [14d9fb88] START, SpmStopVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62), log id: 3cb731d6 >2014-04-28 13:51:16,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-46) [14d9fb88] SpmStopVDSCommand::Stopping SPM on vds 10.34.27.209, pool id 19dcb0b3-4c52-4803-a39e-992b1392da62 >2014-04-28 13:51:16,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-46) [14d9fb88] FINISH, SpmStopVDSCommand, log id: 3cb731d6 >2014-04-28 13:51:16,425 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ResetIrsVDSCommand] (pool-4-thread-46) [14d9fb88] FINISH, ResetIrsVDSCommand, log id: 45946eeb >2014-04-28 13:51:16,426 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SpmStopOnIrsVDSCommand] (pool-4-thread-46) [14d9fb88] FINISH, SpmStopOnIrsVDSCommand, log id: 6d6767c1 >2014-04-28 13:51:16,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (pool-4-thread-46) [14d9fb88] START, DisconnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, vds_spm_id = 1), log id: 4a7c71be >2014-04-28 13:51:16,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (pool-4-thread-46) [14d9fb88] FINISH, DisconnectStoragePoolVDSCommand, log id: 4a7c71be >2014-04-28 13:51:16,663 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-46) [4181d4be] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:51:16,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-46) [4181d4be] START, DisconnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 48bb8365-f600-4eba-8ed7-e823810d8790, connection: /mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 54de30af >2014-04-28 13:51:19,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-46) [4181d4be] FINISH, DisconnectStorageServerVDSCommand, return: {48bb8365-f600-4eba-8ed7-e823810d8790=0}, log id: 54de30af >2014-04-28 13:51:19,086 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [4181d4be] Correlation ID: 14d9fb88, Job ID: fd249b8d-908d-4182-803c-45f6f9195db9, Call Stack: null, Custom Event ID: -1, Message: Storage Domain storage (Data Center Lok) was deactivated by admin@internal >2014-04-28 13:51:27,371 INFO [org.ovirt.engine.core.bll.storage.RemoveStoragePoolCommand] (pool-4-thread-46) [60359a92] Running command: RemoveStoragePoolCommand internal: false. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-28 13:51:27,427 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-48) [4afea028] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:51:27,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [4afea028] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 48bb8365-f600-4eba-8ed7-e823810d8790, connection: /mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1caae667 >2014-04-28 13:51:27,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [4afea028] FINISH, ConnectStorageServerVDSCommand, return: {48bb8365-f600-4eba-8ed7-e823810d8790=0}, log id: 1caae667 >2014-04-28 13:51:27,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) [4afea028] START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, vds_spm_id = 1, masterDomainId = e4aced3b-4048-4151-9a29-3e917e9ed8fd, masterVersion = 1), log id: d1e70e8 >2014-04-28 13:51:30,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) [4afea028] FINISH, ConnectStoragePoolVDSCommand, log id: d1e70e8 >2014-04-28 13:51:30,349 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyStoragePoolVDSCommand] (pool-4-thread-46) [60359a92] START, DestroyStoragePoolVDSCommand( storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, ignoreFailoverLimit = false), log id: 35d44c04 >2014-04-28 13:51:30,353 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (pool-4-thread-46) [42eb45b0] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: 19dcb0b3-4c52-4803-a39e-992b1392da62 Type: StoragePool >2014-04-28 13:51:30,610 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-46) [42eb45b0] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool Lok >2014-04-28 13:51:30,613 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-46) [42eb45b0] starting spm on vds 10.34.27.209, storage pool Lok, prevId -1, LVER 17 >2014-04-28 13:51:30,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-46) [42eb45b0] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, prevId=-1, prevLVER=17, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 3e89c4f9 >2014-04-28 13:51:30,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-46) [42eb45b0] spmStart polling started: taskId = aee75cfd-5d95-46dd-a3cc-57b95ba2413a >2014-04-28 13:51:32,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-46) [42eb45b0] spmStart polling ended: taskId = aee75cfd-5d95-46dd-a3cc-57b95ba2413a task status = finished >2014-04-28 13:51:32,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-46) [42eb45b0] spmStart polling ended, spm status: SPM >2014-04-28 13:51:32,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-46) [42eb45b0] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=aee75cfd-5d95-46dd-a3cc-57b95ba2413a), log id: 178f7d7f >2014-04-28 13:51:32,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-46) [42eb45b0] FINISH, HSMClearTaskVDSCommand, log id: 178f7d7f >2014-04-28 13:51:32,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-46) [42eb45b0] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@449a498a, log id: 3e89c4f9 >2014-04-28 13:51:32,531 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-46) [42eb45b0] Initialize Irs proxy from vds: 10.34.27.209 >2014-04-28 13:51:32,536 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [42eb45b0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host 10.34.27.209 (Address: 10.34.27.209). >2014-04-28 13:51:32,815 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-46) [42eb45b0] IrsProxyData::disposing >2014-04-28 13:51:32,817 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DestroyStoragePoolVDSCommand] (pool-4-thread-46) [42eb45b0] FINISH, DestroyStoragePoolVDSCommand, log id: 35d44c04 >2014-04-28 13:51:32,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (pool-4-thread-46) [42eb45b0] START, FormatStorageDomainVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storageDomainId=e4aced3b-4048-4151-9a29-3e917e9ed8fd), log id: 368b970f >2014-04-28 13:51:35,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.FormatStorageDomainVDSCommand] (pool-4-thread-46) [42eb45b0] FINISH, FormatStorageDomainVDSCommand, log id: 368b970f >2014-04-28 13:51:35,255 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-46) [463f1e3d] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:51:35,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-46) [463f1e3d] START, DisconnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = LOCALFS, connectionList = [{ id: 48bb8365-f600-4eba-8ed7-e823810d8790, connection: /mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: e80bf14 >2014-04-28 13:51:37,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-46) [463f1e3d] FINISH, DisconnectStorageServerVDSCommand, return: {48bb8365-f600-4eba-8ed7-e823810d8790=0}, log id: e80bf14 >2014-04-28 13:51:37,654 INFO [org.ovirt.engine.core.bll.storage.DisconnectStoragePoolAsyncOperation] (pool-4-thread-49) [463f1e3d] Disconnect storage pool treatment vds: 10.34.27.209,pool Lok >2014-04-28 13:51:37,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (pool-4-thread-49) [463f1e3d] START, DisconnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 19dcb0b3-4c52-4803-a39e-992b1392da62, vds_spm_id = 1), log id: 1403e7 >2014-04-28 13:51:37,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStoragePoolVDSCommand] (pool-4-thread-49) [463f1e3d] FINISH, DisconnectStoragePoolVDSCommand, log id: 1403e7 >2014-04-28 13:51:37,916 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [463f1e3d] Correlation ID: 60359a92, Job ID: b22533ec-5095-48f8-8706-721c2f761cb3, Call Stack: null, Custom Event ID: -1, Message: Data Center Lok was removed by admin@internal >2014-04-28 13:51:47,644 INFO [org.ovirt.engine.core.bll.MaintenanceNumberOfVdssCommand] (pool-4-thread-48) [42b807e4] Running command: MaintenanceNumberOfVdssCommand internal: false. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:51:47,646 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [42b807e4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=PreparingForMaintenance, nonOperationalReason=NONE, stopSpmFailureLogged=true), log id: 8e47e7b >2014-04-28 13:51:47,650 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [42b807e4] FINISH, SetVdsStatusVDSCommand, log id: 8e47e7b >2014-04-28 13:51:47,696 INFO [org.ovirt.engine.core.bll.MaintenanceVdsCommand] (pool-4-thread-48) [42b807e4] Running command: MaintenanceVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:51:47,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-48) [42b807e4] Correlation ID: 42b807e4, Job ID: 26b57e11-369b-428f-8a89-55d6c57c50ea, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was switched to Maintenance mode by admin@internal. >2014-04-28 13:51:48,477 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) Updated vds status from Preparing for Maintenance to Maintenance in database, vds = 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 >2014-04-28 13:52:00,081 INFO [org.ovirt.engine.core.bll.ChangeVDSClusterCommand] (ajp-/127.0.0.1:8702-4) [7aa11424] Running command: ChangeVDSClusterCommand internal: false. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS, ID: a13403b3-db7f-4ec4-9b6a-8ec826bd9a15 Type: VdsGroups >2014-04-28 13:52:00,104 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [7aa11424] START, RemoveVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: e8181e >2014-04-28 13:52:00,104 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [7aa11424] vdsManager::disposing >2014-04-28 13:52:00,105 INFO [org.ovirt.engine.core.vdsbroker.RemoveVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [7aa11424] FINISH, RemoveVdsVDSCommand, log id: e8181e >2014-04-28 13:52:00,110 ERROR [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-4) [7aa11424] Cannot get vdsManager for vdsid=53fd133a-9dc6-11e2-abd9-001a4a223991 >2014-04-28 13:52:00,111 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [7aa11424] START, AddVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 60d5d021 >2014-04-28 13:52:00,111 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [7aa11424] AddVds - entered , starting logic to add VDS 53fd133a-9dc6-11e2-abd9-001a4a223991 >2014-04-28 13:52:00,111 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [7aa11424] AddVds - VDS 53fd133a-9dc6-11e2-abd9-001a4a223991 was added, will try to add it to the resource manager >2014-04-28 13:52:00,111 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [7aa11424] Entered VdsManager constructor >2014-04-28 13:52:00,127 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (ajp-/127.0.0.1:8702-4) [7aa11424] Initialize vdsBroker (10.34.27.209,54,321) >2014-04-28 13:52:00,132 INFO [org.ovirt.engine.core.vdsbroker.ResourceManager] (ajp-/127.0.0.1:8702-4) [7aa11424] VDS 53fd133a-9dc6-11e2-abd9-001a4a223991 was added to the Resource Manager >2014-04-28 13:52:00,132 INFO [org.ovirt.engine.core.vdsbroker.AddVdsVDSCommand] (ajp-/127.0.0.1:8702-4) [7aa11424] FINISH, AddVdsVDSCommand, log id: 60d5d021 >2014-04-28 13:52:00,135 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-4) [480e130c] Running command: AddVdsSpmIdCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:52:00,138 INFO [org.ovirt.engine.core.bll.AddVdsSpmIdCommand] (ajp-/127.0.0.1:8702-4) [480e130c] Lock freed to object EngineLock [exclusiveLocks= key: f03e56af-56f8-4150-9517-271bec2305d9 value: REGISTER_VDS >, sharedLocks= ] >2014-04-28 13:52:00,143 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [480e130c] Correlation ID: 7aa11424, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 configuration was updated by admin@internal. >2014-04-28 13:52:00,172 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-04-28 13:52:00,173 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-04-28 13:52:00,180 INFO [org.ovirt.engine.core.bll.UpdateVdsCommand] (ajp-/127.0.0.1:8702-6) [35a77d85] Running command: UpdateVdsCommand internal: false. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:52:00,186 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-04-28 13:52:00,186 WARN [org.ovirt.engine.core.compat.backendcompat.PropertyInfo] (ajp-/127.0.0.1:8702-6) Unable to get value of property: vdsName for class org.ovirt.engine.core.common.businessentities.VdsStatic >2014-04-28 13:52:00,190 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [35a77d85] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-28 13:52:00,198 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-6) [35a77d85] Correlation ID: 35a77d85, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 configuration was updated by admin@internal. >2014-04-28 13:52:01,932 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (ajp-/127.0.0.1:8702-2) [2bda4475] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-28 13:52:01,950 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-46) [2bda4475] Running command: ActivateVdsCommand internal: false. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:52:01,951 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-46) [2bda4475] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-28 13:52:01,951 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-46) [2bda4475] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-28 13:52:01,958 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-46) [2bda4475] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 75b9692f >2014-04-28 13:52:01,961 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-46) [2bda4475] FINISH, SetVdsStatusVDSCommand, log id: 75b9692f >2014-04-28 13:52:01,963 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-46) [2bda4475] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5ea6b3e4 >2014-04-28 13:52:02,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (pool-4-thread-46) [2bda4475] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2d140225 >2014-04-28 13:52:02,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (pool-4-thread-46) [2bda4475] FINISH, GetHardwareInfoVDSCommand, log id: 2d140225 >2014-04-28 13:52:02,738 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (pool-4-thread-46) [4ef7606d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:52:02,739 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (pool-4-thread-46) [1063b6e3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:52:02,740 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (pool-4-thread-46) [1063b6e3] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5ea6b3e4 >2014-04-28 13:52:02,761 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-46) [1063b6e3] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-28 13:52:02,766 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [1063b6e3] Correlation ID: 2bda4475, Job ID: e919695a-c53d-4118-9dcf-f90944151dbf, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was activated by admin@internal. >2014-04-28 13:52:02,769 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (pool-4-thread-46) [1063b6e3] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-28 13:52:03,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 13cf78c0 >2014-04-28 13:52:03,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, GetHardwareInfoVDSCommand, log id: 13cf78c0 >2014-04-28 13:52:03,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 33ca5d5b >2014-04-28 13:52:04,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) FINISH, GetHardwareInfoVDSCommand, log id: 33ca5d5b >2014-04-28 13:52:04,181 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-28 13:52:04,406 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-11) [215c4cc3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-28 13:52:04,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-11) [215c4cc3] START, SetMOMPolicyParametersVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 786f621a >2014-04-28 13:52:04,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SetMOMPolicyParametersVDSCommand] (DefaultQuartzScheduler_Worker-11) [215c4cc3] FINISH, SetMOMPolicyParametersVDSCommand, log id: 786f621a >2014-04-28 13:52:04,737 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [215c4cc3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-28 13:52:04,742 INFO [org.ovirt.engine.core.vdsbroker.VdsManager] (DefaultQuartzScheduler_Worker-11) [215c4cc3] Initializing Host: 10.34.27.209 >2014-04-28 13:52:04,743 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-11) [4b614bd3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:52:04,745 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-28 13:52:16,071 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-24) Setting new tasks map. The map contains now 0 tasks >2014-04-28 13:52:16,071 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-24) Cleared all tasks of pool 19dcb0b3-4c52-4803-a39e-992b1392da62. >2014-04-28 13:52:42,409 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [1d9c25c2] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.27.209:/mnt/storage value: STORAGE_CONNECTION >, sharedLocks= ] >2014-04-28 13:52:42,421 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [1d9c25c2] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:52:42,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [1d9c25c2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 73b6365e >2014-04-28 13:52:42,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [1d9c25c2] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 73b6365e >2014-04-28 13:52:42,787 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [1d9c25c2] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.27.209:/mnt/storage value: STORAGE_CONNECTION >, sharedLocks= ] >2014-04-28 13:52:42,852 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [31a1c4a4] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:52:42,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-3) [31a1c4a4] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: a6d8ea7 >2014-04-28 13:52:43,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-3) [31a1c4a4] FINISH, ConnectStorageServerVDSCommand, return: {854c3686-561c-4079-af7a-2115dab8d0b5=0}, log id: a6d8ea7 >2014-04-28 13:52:43,105 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [31a1c4a4] START, CreateStorageDomainVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@c9369e40, args=10.34.27.209:/mnt/storage), log id: 1a29b35a >2014-04-28 13:52:46,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [31a1c4a4] FINISH, CreateStorageDomainVDSCommand, log id: 1a29b35a >2014-04-28 13:52:46,253 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-3) [31a1c4a4] START, GetStorageDomainStatsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db), log id: 3969e8ee >2014-04-28 13:52:46,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-3) [31a1c4a4] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@d20d33e, log id: 3969e8ee >2014-04-28 13:52:46,504 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [31a1c4a4] Correlation ID: 31a1c4a4, Job ID: 4554aef3-e5b5-4142-a969-f6f0e6a72fe7, Call Stack: null, Custom Event ID: -1, Message: Storage Domain storage was added by admin@internal >2014-04-28 13:52:46,543 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-7) [5c50e654] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:52:46,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [5c50e654] START, DisconnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 625a3a18 >2014-04-28 13:52:48,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-7) [5c50e654] FINISH, DisconnectStorageServerVDSCommand, return: {854c3686-561c-4079-af7a-2115dab8d0b5=0}, log id: 625a3a18 >2014-04-28 13:52:49,025 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-4) [76386407] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: b115b57b-7418-4b60-86b2-64af22fcd1db Type: Storage >2014-04-28 13:52:49,035 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-4) [5abc12e1] Lock Acquired to object EngineLock [exclusiveLocks= key: f03e56af-56f8-4150-9517-271bec2305d9 value: POOL >, sharedLocks= ] >2014-04-28 13:52:49,039 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-4) [5abc12e1] Running command: AddStoragePoolWithStoragesCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-28 13:52:49,097 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (ajp-/127.0.0.1:8702-4) [1388e0f1] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:52:49,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [1388e0f1] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 60d7cf49 >2014-04-28 13:52:49,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-4) [1388e0f1] FINISH, ConnectStorageServerVDSCommand, return: {854c3686-561c-4079-af7a-2115dab8d0b5=0}, log id: 60d7cf49 >2014-04-28 13:52:49,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-4) [1388e0f1] START, CreateStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9, storageType=NFS, storagePoolName=NFS, masterDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, domainsIdList=[b115b57b-7418-4b60-86b2-64af22fcd1db], masterVersion=1), log id: 43c6802 >2014-04-28 13:53:15,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-4) [1388e0f1] FINISH, CreateStoragePoolVDSCommand, log id: 43c6802 >2014-04-28 13:53:15,261 INFO [org.ovirt.engine.core.bll.storage.AddStoragePoolWithStoragesCommand] (ajp-/127.0.0.1:8702-4) [1388e0f1] Lock freed to object EngineLock [exclusiveLocks= key: f03e56af-56f8-4150-9517-271bec2305d9 value: POOL >, sharedLocks= ] >2014-04-28 13:53:15,262 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [77d5827] Lock Acquired to object EngineLock [exclusiveLocks= key: b115b57b-7418-4b60-86b2-64af22fcd1db value: STORAGE >, sharedLocks= ] >2014-04-28 13:53:15,268 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [77d5827] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: b115b57b-7418-4b60-86b2-64af22fcd1db Type: Storage >2014-04-28 13:53:15,273 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [77d5827] Lock freed to object EngineLock [exclusiveLocks= key: b115b57b-7418-4b60-86b2-64af22fcd1db value: STORAGE >, sharedLocks= ] >2014-04-28 13:53:15,273 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [77d5827] ActivateStorage Domain. Before Connect all hosts to pool. Time:4/28/14 1:53 PM >2014-04-28 13:53:15,276 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-49) [7d851e94] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:53:15,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [7d851e94] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 53a8cc50 >2014-04-28 13:53:15,283 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-46) [647e3275] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:53:15,286 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-46) [647e3275] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3016559a >2014-04-28 13:53:15,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [7d851e94] FINISH, ConnectStorageServerVDSCommand, return: {854c3686-561c-4079-af7a-2115dab8d0b5=0}, log id: 53a8cc50 >2014-04-28 13:53:30,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-46) [647e3275] FINISH, ConnectStorageServerVDSCommand, return: {854c3686-561c-4079-af7a-2115dab8d0b5=0}, log id: 3016559a >2014-04-28 13:53:30,909 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-4) [77d5827] START, ActivateStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db), log id: 78a87b84 >2014-04-28 13:53:30,919 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (ajp-/127.0.0.1:8702-4) [59f27575] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-28 13:53:30,925 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [59f27575] Correlation ID: 59f27575, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-04-28 13:53:31,166 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-4) [59f27575] hostFromVds::selectedVds - 10.34.1.251, spmStatus Unknown_Pool, storage pool NFS >2014-04-28 13:53:31,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-4) [59f27575] START, ConnectStoragePoolVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 1, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7cc6f97f >2014-04-28 13:53:36,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (ajp-/127.0.0.1:8702-4) [59f27575] FINISH, ConnectStoragePoolVDSCommand, log id: 7cc6f97f >2014-04-28 13:53:36,481 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-4) [59f27575] hostFromVds::Connected host to pool - selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-28 13:53:36,488 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-4) [59f27575] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER -1 >2014-04-28 13:53:36,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-4) [59f27575] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=-1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 652bbb32 >2014-04-28 13:53:36,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-4) [59f27575] spmStart polling started: taskId = 10c98489-3503-485d-8859-7a738727dfbf >2014-04-28 13:54:04,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-4) [59f27575] spmStart polling ended: taskId = 10c98489-3503-485d-8859-7a738727dfbf task status = finished >2014-04-28 13:54:04,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-4) [59f27575] spmStart polling ended, spm status: SPM >2014-04-28 13:54:04,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-4) [59f27575] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=10c98489-3503-485d-8859-7a738727dfbf), log id: 289961f1 >2014-04-28 13:54:04,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (ajp-/127.0.0.1:8702-4) [59f27575] FINISH, HSMClearTaskVDSCommand, log id: 289961f1 >2014-04-28 13:54:04,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (ajp-/127.0.0.1:8702-4) [59f27575] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45c54aab, log id: 652bbb32 >2014-04-28 13:54:04,712 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (ajp-/127.0.0.1:8702-4) [59f27575] Initialize Irs proxy from vds: 10.34.1.251 >2014-04-28 13:54:04,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [59f27575] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host 10.34.1.251 (Address: 10.34.1.251). >2014-04-28 13:54:04,965 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-4) [59f27575] FINISH, ActivateStorageDomainVDSCommand, log id: 78a87b84 >2014-04-28 13:54:04,967 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [59f27575] ActivateStorage Domain. After Connect all hosts to pool. Time:4/28/14 1:54 PM >2014-04-28 13:54:04,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) [59f27575] START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 21b4c4f5 >2014-04-28 13:54:04,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) [59f27575] START, ConnectStoragePoolVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 1, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4e2504a0 >2014-04-28 13:54:05,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) [59f27575] FINISH, ConnectStoragePoolVDSCommand, log id: 4e2504a0 >2014-04-28 13:54:07,668 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) [59f27575] FINISH, ConnectStoragePoolVDSCommand, log id: 21b4c4f5 >2014-04-28 13:54:07,675 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [59f27575] ActivateStorage Domain. After change storage pool status in vds. Time:4/28/14 1:54 PM >2014-04-28 13:54:07,683 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [59f27575] Correlation ID: 77d5827, Call Stack: null, Custom Event ID: -1, Message: Storage Domain storage (Data Center NFS) was activated by admin@internal >2014-04-28 13:54:07,689 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [59f27575] Correlation ID: 5abc12e1, Call Stack: null, Custom Event ID: -1, Message: Storage Domains were attached to Data Center NFS by admin@internal >2014-04-28 13:54:07,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [59f27575] Correlation ID: 76386407, Job ID: a60351d8-837e-4fc4-b2df-ea1b6f51a24c, Call Stack: null, Custom Event ID: -1, Message: Storage Domain storage was attached to Data Center NFS by admin@internal >2014-04-28 13:54:33,174 INFO [org.ovirt.engine.core.bll.RemoveVdsGroupCommand] (pool-4-thread-49) [15440c0e] Running command: RemoveVdsGroupCommand internal: false. Entities affected : ID: 481e4b4c-9dc6-11e2-ab66-001a4a223991 Type: VdsGroups >2014-04-28 13:54:33,201 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-49) [15440c0e] Correlation ID: 15440c0e, Call Stack: null, Custom Event ID: -1, Message: Host cluster Local was removed by admin@internal >2014-04-28 13:54:45,081 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (pool-4-thread-50) [4f6cbc8] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 Type: Storage >2014-04-28 13:54:45,091 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-49) [6e1a64ba] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:54:45,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [6e1a64ba] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 31aae5eb >2014-04-28 13:54:45,094 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-46) [31be80c2] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:54:45,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-46) [31be80c2] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 386a9729 >2014-04-28 13:54:45,437 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-46) [31be80c2] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 386a9729 >2014-04-28 13:54:45,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-49) [6e1a64ba] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 31aae5eb >2014-04-28 13:54:45,625 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (pool-4-thread-50) [4f6cbc8] START, AttachStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = 6ba7421b-4b80-4aff-ba01-1c08d3ef4619), log id: f600a7a >2014-04-28 13:54:57,103 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (pool-4-thread-46) [4adbf28] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: c0532ddb-5faf-4ab6-b93f-c490c588cae4 Type: Storage >2014-04-28 13:54:57,108 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-47) [55a0a1b9] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:54:57,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [55a0a1b9] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1534f71d >2014-04-28 13:54:57,116 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-48) [12313209] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:54:57,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [12313209] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 544b4e29 >2014-04-28 13:54:57,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [55a0a1b9] FINISH, ConnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 1534f71d >2014-04-28 13:54:57,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [12313209] FINISH, ConnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 544b4e29 >2014-04-28 13:54:57,424 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (pool-4-thread-46) [4adbf28] START, AttachStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4), log id: 768e1dfa >2014-04-28 13:55:09,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (pool-4-thread-50) [4f6cbc8] FINISH, AttachStorageDomainVDSCommand, log id: f600a7a >2014-04-28 13:55:09,534 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-50) [3a1985aa] Lock Acquired to object EngineLock [exclusiveLocks= key: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 value: STORAGE >, sharedLocks= ] >2014-04-28 13:55:09,544 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-50) [3a1985aa] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 Type: Storage >2014-04-28 13:55:09,548 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-50) [3a1985aa] Lock freed to object EngineLock [exclusiveLocks= key: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 value: STORAGE >, sharedLocks= ] >2014-04-28 13:55:09,548 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-50) [3a1985aa] ActivateStorage Domain. Before Connect all hosts to pool. Time:4/28/14 1:55 PM >2014-04-28 13:55:09,550 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-45) [69b26879] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:55:09,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-45) [69b26879] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 10c061c7 >2014-04-28 13:55:09,563 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-48) [3f88d6cf] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:55:09,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [3f88d6cf] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50836db4 >2014-04-28 13:55:09,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-45) [69b26879] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 10c061c7 >2014-04-28 13:55:09,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [3f88d6cf] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 50836db4 >2014-04-28 13:55:09,800 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-50) [3a1985aa] START, ActivateStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = 6ba7421b-4b80-4aff-ba01-1c08d3ef4619), log id: 4a347785 >2014-04-28 13:55:16,440 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (pool-4-thread-46) [4adbf28] FINISH, AttachStorageDomainVDSCommand, log id: 768e1dfa >2014-04-28 13:55:16,444 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [5a88b273] Lock Acquired to object EngineLock [exclusiveLocks= key: c0532ddb-5faf-4ab6-b93f-c490c588cae4 value: STORAGE >, sharedLocks= ] >2014-04-28 13:55:16,452 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [5a88b273] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: c0532ddb-5faf-4ab6-b93f-c490c588cae4 Type: Storage >2014-04-28 13:55:16,459 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [5a88b273] Lock freed to object EngineLock [exclusiveLocks= key: c0532ddb-5faf-4ab6-b93f-c490c588cae4 value: STORAGE >, sharedLocks= ] >2014-04-28 13:55:16,459 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [5a88b273] ActivateStorage Domain. Before Connect all hosts to pool. Time:4/28/14 1:55 PM >2014-04-28 13:55:16,462 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-44) [1e4be5bd] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:55:16,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-44) [1e4be5bd] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6115db83 >2014-04-28 13:55:16,465 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-43) [12b41aba] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-28 13:55:16,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-43) [12b41aba] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4e217a69 >2014-04-28 13:55:16,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-43) [12b41aba] FINISH, ConnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 4e217a69 >2014-04-28 13:55:16,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-44) [1e4be5bd] FINISH, ConnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 6115db83 >2014-04-28 13:55:16,916 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-46) [5a88b273] START, ActivateStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4), log id: 2d91114f >2014-04-28 13:55:20,543 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-46) [5a88b273] FINISH, ActivateStorageDomainVDSCommand, log id: 2d91114f >2014-04-28 13:55:20,543 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [5a88b273] ActivateStorage Domain. After Connect all hosts to pool. Time:4/28/14 1:55 PM >2014-04-28 13:55:20,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-44) [5a88b273] START, RefreshStoragePoolVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 6e5ab97a >2014-04-28 13:55:20,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-45) [5a88b273] START, RefreshStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 70013653 >2014-04-28 13:55:21,215 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (pool-4-thread-50) [3a1985aa] FINISH, ActivateStorageDomainVDSCommand, log id: 4a347785 >2014-04-28 13:55:21,215 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-50) [3a1985aa] ActivateStorage Domain. After Connect all hosts to pool. Time:4/28/14 1:55 PM >2014-04-28 13:55:21,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-48) [3a1985aa] START, RefreshStoragePoolVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 1348ca >2014-04-28 13:55:21,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [3a1985aa] START, RefreshStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 6a985eaf >2014-04-28 13:55:21,222 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-44) [5a88b273] FINISH, RefreshStoragePoolVDSCommand, log id: 6e5ab97a >2014-04-28 13:55:21,222 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (pool-4-thread-44) [5a88b273] Refreshed vds 10.34.1.251 in pool NFS >2014-04-28 13:55:21,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-48) [3a1985aa] FINISH, RefreshStoragePoolVDSCommand, log id: 1348ca >2014-04-28 13:55:21,446 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (pool-4-thread-48) [3a1985aa] Refreshed vds 10.34.1.251 in pool NFS >2014-04-28 13:55:25,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [3a1985aa] FINISH, RefreshStoragePoolVDSCommand, log id: 6a985eaf >2014-04-28 13:55:25,471 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (pool-4-thread-49) [3a1985aa] Refreshed vds 10.34.27.209 in pool NFS >2014-04-28 13:55:25,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-45) [5a88b273] FINISH, RefreshStoragePoolVDSCommand, log id: 70013653 >2014-04-28 13:55:25,474 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (pool-4-thread-45) [5a88b273] Refreshed vds 10.34.27.209 in pool NFS >2014-04-28 13:55:25,476 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-50) [3a1985aa] ActivateStorage Domain. After change storage pool status in vds. Time:4/28/14 1:55 PM >2014-04-28 13:55:25,477 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) [3a1985aa] START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 5088173f >2014-04-28 13:55:25,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) [3a1985aa] START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 16ae5267 >2014-04-28 13:55:25,479 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (pool-4-thread-46) [5a88b273] ActivateStorage Domain. After change storage pool status in vds. Time:4/28/14 1:55 PM >2014-04-28 13:55:25,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [5a88b273] Correlation ID: 5a88b273, Call Stack: null, Custom Event ID: -1, Message: Storage Domain gal (Data Center NFS) was activated by admin@internal >2014-04-28 13:55:25,488 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) [3a1985aa] Correlation ID: 3a1985aa, Call Stack: null, Custom Event ID: -1, Message: Storage Domain iso (Data Center NFS) was activated by admin@internal >2014-04-28 13:55:25,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [5a88b273] Correlation ID: 4adbf28, Job ID: 7e676722-28c5-4db9-ab03-f4571950d0ea, Call Stack: null, Custom Event ID: -1, Message: Storage Domain gal was attached to Data Center NFS by admin@internal >2014-04-28 13:55:25,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) [3a1985aa] Correlation ID: 4f6cbc8, Job ID: a1597458-c920-4dc7-a258-06dc48a96dc0, Call Stack: null, Custom Event ID: -1, Message: Storage Domain iso was attached to Data Center NFS by admin@internal >2014-04-28 13:55:25,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) [3a1985aa] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 16ae5267 >2014-04-28 13:55:25,727 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) [3a1985aa] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5088173f >2014-04-28 13:55:25,733 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetFloppyListVDSCommand] (pool-4-thread-45) [3a1985aa] START, GetFloppyListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: f30f35a >2014-04-28 13:55:25,966 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetFloppyListVDSCommand] (pool-4-thread-45) [3a1985aa] FINISH, GetFloppyListVDSCommand, return: [], log id: f30f35a >2014-04-28 13:55:40,475 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (ajp-/127.0.0.1:8702-7) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 6c5d20bf >2014-04-28 13:55:40,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (ajp-/127.0.0.1:8702-7) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 5611132d >2014-04-28 13:55:40,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (ajp-/127.0.0.1:8702-7) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5611132d >2014-04-28 13:55:40,774 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (ajp-/127.0.0.1:8702-7) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6c5d20bf >2014-04-28 13:55:40,927 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetFloppyListVDSCommand] (ajp-/127.0.0.1:8702-7) START, GetFloppyListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 3b01a8cd >2014-04-28 13:55:41,188 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetFloppyListVDSCommand] (ajp-/127.0.0.1:8702-7) FINISH, GetFloppyListVDSCommand, return: [], log id: 3b01a8cd >2014-04-28 13:55:41,192 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-7) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Refresh image list succeeded for domain(s): iso (All file type) >2014-04-28 13:55:47,255 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-6) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 144002ab >2014-04-28 13:55:47,494 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-6) FINISH, GetVmsInfoVDSCommand, log id: 144002ab >2014-04-28 13:55:50,024 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-7) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 69801f81 >2014-04-28 13:55:50,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-7) FINISH, GetVmsInfoVDSCommand, log id: 69801f81 >2014-04-28 13:55:55,285 INFO [org.ovirt.engine.core.bll.ImportVmCommand] (ajp-/127.0.0.1:8702-5) [3124dd06] Lock Acquired to object EngineLock [exclusiveLocks= key: Gal value: VM_NAME >, sharedLocks= key: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 value: REMOTE_VM >] >2014-04-28 13:55:55,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-5) [3124dd06] START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 29cbded7 >2014-04-28 13:55:55,524 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-5) [3124dd06] FINISH, GetVmsInfoVDSCommand, log id: 29cbded7 >2014-04-28 13:55:55,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-5) [3124dd06] START, GetImageDomainsListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, imageGroupId = 431c414f-83e8-4ad7-a08b-050ee71f9784), log id: 49e5c970 >2014-04-28 13:55:56,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-5) [3124dd06] FINISH, GetImageDomainsListVDSCommand, return: [], log id: 49e5c970 >2014-04-28 13:55:56,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp-/127.0.0.1:8702-5) [3124dd06] START, DoesImageExistVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, imageGroupId = 431c414f-83e8-4ad7-a08b-050ee71f9784, imageId = 2b3a5ce1-e07f-4d7e-bf3a-213d7587cfbb), log id: 7c969986 >2014-04-28 13:55:56,395 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DoesImageExistVDSCommand] (ajp-/127.0.0.1:8702-5) [3124dd06] FINISH, DoesImageExistVDSCommand, return: true, log id: 7c969986 >2014-04-28 13:55:56,401 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-5) [3124dd06] START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 33d97e02 >2014-04-28 13:55:56,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-5) [3124dd06] FINISH, GetVmsInfoVDSCommand, log id: 33d97e02 >2014-04-28 13:55:57,019 INFO [org.ovirt.engine.core.bll.ImportVmCommand] (pool-4-thread-45) [3124dd06] Running command: ImportVmCommand internal: false. Entities affected : ID: b115b57b-7418-4b60-86b2-64af22fcd1db Type: Storage >2014-04-28 13:55:57,072 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-6) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 3a9ad45c >2014-04-28 13:55:57,178 INFO [org.ovirt.engine.core.bll.CopyImageGroupCommand] (pool-4-thread-45) [b8511c7] Running command: CopyImageGroupCommand internal: true. Entities affected : ID: b115b57b-7418-4b60-86b2-64af22fcd1db Type: Storage >2014-04-28 13:55:57,284 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-45) [b8511c7] START, CopyImageVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, imageGroupId = 431c414f-83e8-4ad7-a08b-050ee71f9784, imageId = 2b3a5ce1-e07f-4d7e-bf3a-213d7587cfbb, dstImageGroupId = 431c414f-83e8-4ad7-a08b-050ee71f9784, vmId = b066ff4b-970f-4305-94a4-1f32f7fa6fe2, dstImageId = 2b3a5ce1-e07f-4d7e-bf3a-213d7587cfbb, imageDescription = , dstStorageDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, copyVolumeType = LeafVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = true), log id: 9fef3b1 >2014-04-28 13:55:57,303 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-6) FINISH, GetVmsInfoVDSCommand, log id: 3a9ad45c >2014-04-28 13:55:57,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-45) [b8511c7] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID >2014-04-28 13:55:57,305 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-45) [b8511c7] -- copyImage parameters: > sdUUID=c0532ddb-5faf-4ab6-b93f-c490c588cae4 > spUUID=f03e56af-56f8-4150-9517-271bec2305d9 > vmGUID=b066ff4b-970f-4305-94a4-1f32f7fa6fe2 > srcImageGUID=431c414f-83e8-4ad7-a08b-050ee71f9784 > srcVolUUID=2b3a5ce1-e07f-4d7e-bf3a-213d7587cfbb > dstImageGUID=431c414f-83e8-4ad7-a08b-050ee71f9784 > dstVolUUID=2b3a5ce1-e07f-4d7e-bf3a-213d7587cfbb > descr= > dstSdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db >2014-04-28 13:55:57,946 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-45) [b8511c7] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 9fef3b1 >2014-04-28 13:55:58,154 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-45) [b8511c7] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command 0e66a3b7-79b1-4182-a8a3-98b9352a21fd >2014-04-28 13:55:58,154 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-4-thread-45) [b8511c7] CommandMultiAsyncTasks::AttachTask: Attaching task 7e3446b1-d59e-442d-82f7-3abb535daa63 to command 0e66a3b7-79b1-4182-a8a3-98b9352a21fd. >2014-04-28 13:55:58,227 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-45) [b8511c7] Adding task 7e3446b1-d59e-442d-82f7-3abb535daa63 (Parent Command ImportVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-04-28 13:55:58,364 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-45) [b8511c7] START, SetVmStatusVDSCommand( vmId = b066ff4b-970f-4305-94a4-1f32f7fa6fe2, status = ImageLocked), log id: 298ba034 >2014-04-28 13:55:58,368 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-45) [b8511c7] FINISH, SetVmStatusVDSCommand, log id: 298ba034 >2014-04-28 13:55:58,443 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-45) [b8511c7] Correlation ID: 3124dd06, Job ID: ae117c55-0200-41d7-a66f-b19aea975154, Call Stack: null, Custom Event ID: -1, Message: Starting to import Vm Gal to Data Center NFS, Cluster NFS >2014-04-28 13:55:58,443 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-45) [b8511c7] BaseAsyncTask::StartPollingTask: Starting to poll task 7e3446b1-d59e-442d-82f7-3abb535daa63. >2014-04-28 13:56:00,667 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-75) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-04-28 13:56:00,896 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-75) SPMAsyncTask::PollTask: Polling task 7e3446b1-d59e-442d-82f7-3abb535daa63 (Parent Command ImportVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-04-28 13:56:00,896 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-75) Finished polling Tasks, will poll again in 10 seconds. >2014-04-28 14:01:51,923 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Used Network resources of host 10.34.27.209 [98%] exceeded defined threshold [95%]. >2014-04-28 14:20:26,958 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-97) SPMAsyncTask::PollTask: Polling task 7e3446b1-d59e-442d-82f7-3abb535daa63 (Parent Command ImportVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-04-28 14:20:26,964 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-97) BaseAsyncTask::OnTaskEndSuccess: Task 7e3446b1-d59e-442d-82f7-3abb535daa63 (Parent Command ImportVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-04-28 14:20:26,964 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-97) CommandAsyncTask::EndActionIfNecessary: All tasks of command 0e66a3b7-79b1-4182-a8a3-98b9352a21fd has ended -> executing EndAction >2014-04-28 14:20:26,964 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-97) CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: 0e66a3b7-79b1-4182-a8a3-98b9352a21fd): calling EndAction . >2014-04-28 14:20:26,964 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to EndAction ImportVm, executionIndex: 0 >2014-04-28 14:20:26,972 INFO [org.ovirt.engine.core.bll.ImportVmCommand] (pool-4-thread-50) Ending command successfully: org.ovirt.engine.core.bll.ImportVmCommand >2014-04-28 14:20:26,978 INFO [org.ovirt.engine.core.bll.CopyImageGroupCommand] (pool-4-thread-50) [b8511c7] Ending command successfully: org.ovirt.engine.core.bll.CopyImageGroupCommand >2014-04-28 14:20:26,983 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [b8511c7] START, GetImageInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, imageGroupId = 431c414f-83e8-4ad7-a08b-050ee71f9784, imageId = 2b3a5ce1-e07f-4d7e-bf3a-213d7587cfbb), log id: 740db3fc >2014-04-28 14:20:27,271 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [b8511c7] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@e70a4a70, log id: 740db3fc >2014-04-28 14:20:27,281 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-50) START, SetVmStatusVDSCommand( vmId = b066ff4b-970f-4305-94a4-1f32f7fa6fe2, status = Down), log id: 72e72574 >2014-04-28 14:20:27,283 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-50) FINISH, SetVmStatusVDSCommand, log id: 72e72574 >2014-04-28 14:20:27,283 INFO [org.ovirt.engine.core.bll.ImportVmCommand] (pool-4-thread-50) Lock freed to object EngineLock [exclusiveLocks= key: Gal value: VM_NAME >, sharedLocks= key: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 value: REMOTE_VM >] >2014-04-28 14:20:27,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) Correlation ID: 3124dd06, Job ID: ae117c55-0200-41d7-a66f-b19aea975154, Call Stack: null, Custom Event ID: -1, Message: Vm Gal was imported successfully to Data Center NFS, Cluster NFS >2014-04-28 14:20:27,290 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type ImportVm completed, handling the result. >2014-04-28 14:20:27,290 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type ImportVm succeeded, clearing tasks. >2014-04-28 14:20:27,294 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) SPMAsyncTask::ClearAsyncTask: Attempting to clear task 7e3446b1-d59e-442d-82f7-3abb535daa63 >2014-04-28 14:20:27,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) START, SPMClearTaskVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, taskId = 7e3446b1-d59e-442d-82f7-3abb535daa63), log id: 21dbc9e5 >2014-04-28 14:20:27,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=7e3446b1-d59e-442d-82f7-3abb535daa63), log id: 64a3eac0 >2014-04-28 14:20:27,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) FINISH, HSMClearTaskVDSCommand, log id: 64a3eac0 >2014-04-28 14:20:27,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) FINISH, SPMClearTaskVDSCommand, log id: 21dbc9e5 >2014-04-28 14:20:27,543 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) BaseAsyncTask::RemoveTaskFromDB: Removed task 7e3446b1-d59e-442d-82f7-3abb535daa63 from DataBase >2014-04-28 14:20:27,544 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity 0e66a3b7-79b1-4182-a8a3-98b9352a21fd >2014-04-28 14:21:27,944 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-60) Setting new tasks map. The map contains now 0 tasks >2014-04-28 14:21:27,944 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-60) Cleared all tasks of pool f03e56af-56f8-4150-9517-271bec2305d9. >2014-04-28 14:36:04,939 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-20) Start refreshing all users data >2014-04-28 14:36:06,191 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-28 14:36:06,192 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center NFS >2014-04-28 14:36:06,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (DefaultQuartzScheduler_Worker-98) START, UpdateVMVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 1259054d >2014-04-28 14:36:06,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, UpdateVMVDSCommand, log id: 1259054d >2014-04-28 14:36:06,541 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center NFS >2014-04-28 14:36:06,541 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center NFS >2014-04-28 14:36:06,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center NFS >2014-04-28 14:36:06,543 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-28 14:36:06,544 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-28 15:36:04,941 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-42) Start refreshing all users data >2014-04-28 15:36:06,544 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-04-28 15:36:06,545 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VM OVFs in Data Center NFS >2014-04-28 15:36:06,555 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated VM OVFs in Data Center NFS >2014-04-28 15:36:06,555 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update template OVFs in Data Center NFS >2014-04-28 15:36:06,556 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated templates OVFs in Data Center NFS >2014-04-28 15:36:06,556 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-28 15:36:06,557 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-28 15:42:42,489 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 3830ee62 >2014-04-28 15:42:42,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 147280c0 >2014-04-28 15:42:42,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 147280c0 >2014-04-28 15:42:42,743 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3830ee62 >2014-04-28 15:42:42,748 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-45) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 16:36:04,943 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-70) Start refreshing all users data >2014-04-28 16:36:06,557 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VMs/Templates Ovf. >2014-04-28 16:36:06,558 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update VM OVFs in Data Center NFS >2014-04-28 16:36:06,559 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated VM OVFs in Data Center NFS >2014-04-28 16:36:06,559 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to update template OVFs in Data Center NFS >2014-04-28 16:36:06,560 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully updated templates OVFs in Data Center NFS >2014-04-28 16:36:06,560 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-28 16:36:06,560 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-76) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-28 16:42:42,754 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 6ec2abb3 >2014-04-28 16:42:42,755 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 2e96ad94 >2014-04-28 16:42:43,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e96ad94 >2014-04-28 16:42:43,016 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ec2abb3 >2014-04-28 16:42:43,021 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 16:48:15,959 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space.gal domain has 5 GB of free space >2014-04-28 16:48:15,969 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space.storage domain has 5 GB of free space >2014-04-28 16:48:15,976 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space.iso domain has 5 GB of free space >2014-04-28 16:49:05,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space. Host 10.34.27.209 has less than 1000 MB of free space left on: /var/log/core, /tmp, /var/log, /var/run/vdsm/. >2014-04-28 16:49:08,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. gal domain has 0 GB of free space >2014-04-28 16:49:08,629 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. iso domain has 0 GB of free space >2014-04-28 16:49:19,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. storage domain has 0 GB of free space >2014-04-28 16:49:21,854 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. Host 10.34.27.209 has less than 500 MB of free space left on: /var/log/core, /tmp, /var/log, /var/run/vdsm/. >2014-04-28 17:05:50,277 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. gal domain has 0 GB of free space >2014-04-28 17:05:50,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. storage domain has 0 GB of free space >2014-04-28 17:05:50,292 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. iso domain has 0 GB of free space >2014-04-28 17:06:05,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. Host 10.34.27.209 has less than 500 MB of free space left on: /var/log/core, /tmp, /var/log, /var/run/vdsm/. >2014-04-28 17:09:39,197 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-4) Running command: LoginAdminUserCommand internal: false. >2014-04-28 17:09:39,237 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-28 17:09:39,286 INFO [org.ovirt.engine.api.restapi.security.auth.LoginValidator] (ajp-/127.0.0.1:8702-1) Validating session failed, reason: Session does not exist. >2014-04-28 17:09:55,243 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-9) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 7d016d2f >2014-04-28 17:09:55,520 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-9) FINISH, GetVmsInfoVDSCommand, log id: 7d016d2f >2014-04-28 17:09:56,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-4) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 7ae8e835 >2014-04-28 17:09:57,182 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-4) FINISH, GetVmsInfoVDSCommand, log id: 7ae8e835 >2014-04-28 17:10:19,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-4) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 16197b26 >2014-04-28 17:10:19,713 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-4) FINISH, GetVmsInfoVDSCommand, log id: 16197b26 >2014-04-28 17:10:20,724 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-9) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 77b2f5cb >2014-04-28 17:10:20,961 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-9) FINISH, GetVmsInfoVDSCommand, log id: 77b2f5cb >2014-04-28 17:10:54,945 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-9) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 2dbffb98 >2014-04-28 17:10:55,178 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-9) FINISH, GetVmsInfoVDSCommand, log id: 2dbffb98 >2014-04-28 17:10:56,183 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-7) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 11079aea >2014-04-28 17:10:56,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-7) FINISH, GetVmsInfoVDSCommand, log id: 11079aea >2014-04-28 17:11:59,314 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-11) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 262fa3bb >2014-04-28 17:11:59,547 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-11) FINISH, GetVmsInfoVDSCommand, log id: 262fa3bb >2014-04-28 17:12:00,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-4) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 49bd7051 >2014-04-28 17:12:01,110 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-4) FINISH, GetVmsInfoVDSCommand, log id: 49bd7051 >2014-04-28 17:17:46,917 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-3) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 74bd1d48 >2014-04-28 17:17:52,330 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-3) FINISH, GetVmsInfoVDSCommand, log id: 74bd1d48 >2014-04-28 17:20:28,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [35c89394] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Used Network resources of host 10.34.27.209 [100%] exceeded defined threshold [95%]. >2014-04-28 17:36:04,946 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-40) Start refreshing all users data >2014-04-28 17:36:06,560 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VMs/Templates Ovf. >2014-04-28 17:36:06,561 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update VM OVFs in Data Center NFS >2014-04-28 17:36:06,562 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated VM OVFs in Data Center NFS >2014-04-28 17:36:06,562 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to update template OVFs in Data Center NFS >2014-04-28 17:36:06,563 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully updated templates OVFs in Data Center NFS >2014-04-28 17:36:06,563 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-28 17:36:06,563 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-28 17:42:43,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 206d5333 >2014-04-28 17:42:43,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 697149e6 >2014-04-28 17:42:43,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 697149e6 >2014-04-28 17:42:43,310 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 206d5333 >2014-04-28 17:42:43,315 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 17:50:36,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Used Network resources of host 10.34.27.209 [98%] exceeded defined threshold [95%]. >2014-04-28 18:06:06,623 INFO [org.ovirt.engine.core.bll.LogoutUserCommand] (ajp-/127.0.0.1:8702-1) [6a52955b] Running command: LogoutUserCommand internal: false. >2014-04-28 18:06:06,640 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [6a52955b] Correlation ID: 6a52955b, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged out. >2014-04-28 18:36:04,948 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-04-28 18:36:06,564 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VMs/Templates Ovf. >2014-04-28 18:36:06,565 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update VM OVFs in Data Center NFS >2014-04-28 18:36:06,565 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated VM OVFs in Data Center NFS >2014-04-28 18:36:06,566 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to update template OVFs in Data Center NFS >2014-04-28 18:36:06,566 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully updated templates OVFs in Data Center NFS >2014-04-28 18:36:06,566 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-28 18:36:06,567 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-84) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-28 18:42:43,317 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: f92b6e8 >2014-04-28 18:42:43,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 69264154 >2014-04-28 18:42:43,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 69264154 >2014-04-28 18:42:43,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: f92b6e8 >2014-04-28 18:42:43,882 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 19:36:04,951 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Start refreshing all users data >2014-04-28 19:36:06,567 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VMs/Templates Ovf. >2014-04-28 19:36:06,568 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update VM OVFs in Data Center NFS >2014-04-28 19:36:06,573 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated VM OVFs in Data Center NFS >2014-04-28 19:36:06,573 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to update template OVFs in Data Center NFS >2014-04-28 19:36:06,574 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully updated templates OVFs in Data Center NFS >2014-04-28 19:36:06,574 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-28 19:36:06,575 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-28 19:42:43,885 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 776dbc2 >2014-04-28 19:42:43,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: c55a162 >2014-04-28 19:42:44,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-47) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: c55a162 >2014-04-28 19:42:44,125 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-47) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 776dbc2 >2014-04-28 19:42:44,130 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-47) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 20:36:04,955 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-36) Start refreshing all users data >2014-04-28 20:36:06,575 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VMs/Templates Ovf. >2014-04-28 20:36:06,576 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update VM OVFs in Data Center NFS >2014-04-28 20:36:06,577 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated VM OVFs in Data Center NFS >2014-04-28 20:36:06,577 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to update template OVFs in Data Center NFS >2014-04-28 20:36:06,577 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully updated templates OVFs in Data Center NFS >2014-04-28 20:36:06,577 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-28 20:36:06,578 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-28 20:42:44,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 576cb453 >2014-04-28 20:42:44,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 21b2e1fc >2014-04-28 20:42:44,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 21b2e1fc >2014-04-28 20:42:44,376 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 576cb453 >2014-04-28 20:42:44,380 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-45) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 21:36:04,956 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-81) Start refreshing all users data >2014-04-28 21:36:06,578 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VMs/Templates Ovf. >2014-04-28 21:36:06,579 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update VM OVFs in Data Center NFS >2014-04-28 21:36:06,580 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated VM OVFs in Data Center NFS >2014-04-28 21:36:06,580 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to update template OVFs in Data Center NFS >2014-04-28 21:36:06,581 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully updated templates OVFs in Data Center NFS >2014-04-28 21:36:06,581 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-28 21:36:06,581 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-94) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-28 21:42:44,387 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [431634a] START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 47f328f6 >2014-04-28 21:42:44,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [431634a] START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 2d065361 >2014-04-28 21:42:44,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) [431634a] FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2d065361 >2014-04-28 21:42:44,631 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) [431634a] FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 47f328f6 >2014-04-28 21:42:44,635 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) [431634a] Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 22:36:04,958 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-20) Start refreshing all users data >2014-04-28 22:36:06,581 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VMs/Templates Ovf. >2014-04-28 22:36:06,582 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update VM OVFs in Data Center NFS >2014-04-28 22:36:06,583 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated VM OVFs in Data Center NFS >2014-04-28 22:36:06,583 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to update template OVFs in Data Center NFS >2014-04-28 22:36:06,583 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully updated templates OVFs in Data Center NFS >2014-04-28 22:36:06,583 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-28 22:36:06,584 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-98) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-28 22:42:44,638 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 162f9867 >2014-04-28 22:42:44,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: ebe9d92 >2014-04-28 22:42:44,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: ebe9d92 >2014-04-28 22:42:44,881 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 162f9867 >2014-04-28 22:42:44,886 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-45) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-28 23:36:04,960 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) Start refreshing all users data >2014-04-28 23:36:06,585 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) [35c89394] Attempting to update VMs/Templates Ovf. >2014-04-28 23:36:06,586 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) [35c89394] Attempting to update VM OVFs in Data Center NFS >2014-04-28 23:36:06,586 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) [35c89394] Successfully updated VM OVFs in Data Center NFS >2014-04-28 23:36:06,587 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) [35c89394] Attempting to update template OVFs in Data Center NFS >2014-04-28 23:36:06,587 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) [35c89394] Successfully updated templates OVFs in Data Center NFS >2014-04-28 23:36:06,587 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) [35c89394] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-28 23:36:06,588 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-50) [35c89394] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-28 23:42:44,887 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 6eeaaf15 >2014-04-28 23:42:44,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 718cc496 >2014-04-28 23:42:45,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 718cc496 >2014-04-28 23:42:45,132 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6eeaaf15 >2014-04-28 23:42:45,137 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-29 00:36:04,962 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-97) Start refreshing all users data >2014-04-29 00:36:06,588 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VMs/Templates Ovf. >2014-04-29 00:36:06,589 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update VM OVFs in Data Center NFS >2014-04-29 00:36:06,590 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated VM OVFs in Data Center NFS >2014-04-29 00:36:06,590 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to update template OVFs in Data Center NFS >2014-04-29 00:36:06,591 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully updated templates OVFs in Data Center NFS >2014-04-29 00:36:06,591 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 00:36:06,592 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-66) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 00:42:45,141 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 676011bf >2014-04-29 00:42:45,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 542d479e >2014-04-29 00:42:45,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 542d479e >2014-04-29 00:42:45,399 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 676011bf >2014-04-29 00:42:45,403 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-29 01:36:04,964 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-56) [21697a9d] Start refreshing all users data >2014-04-29 01:36:06,592 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update VMs/Templates Ovf. >2014-04-29 01:36:06,593 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update VM OVFs in Data Center NFS >2014-04-29 01:36:06,594 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully updated VM OVFs in Data Center NFS >2014-04-29 01:36:06,594 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to update template OVFs in Data Center NFS >2014-04-29 01:36:06,594 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully updated templates OVFs in Data Center NFS >2014-04-29 01:36:06,595 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 01:36:06,595 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-33) [431634a] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 01:42:45,406 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-46) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 45252e2a >2014-04-29 01:42:45,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-46) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: bf9eadd >2014-04-29 01:42:45,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-46) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: bf9eadd >2014-04-29 01:42:45,681 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-46) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 45252e2a >2014-04-29 01:42:45,687 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-46) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-29 02:36:04,965 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-94) Start refreshing all users data >2014-04-29 02:36:06,595 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VMs/Templates Ovf. >2014-04-29 02:36:06,596 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update VM OVFs in Data Center NFS >2014-04-29 02:36:06,597 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated VM OVFs in Data Center NFS >2014-04-29 02:36:06,597 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to update template OVFs in Data Center NFS >2014-04-29 02:36:06,598 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully updated templates OVFs in Data Center NFS >2014-04-29 02:36:06,598 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 02:36:06,598 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-5) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 02:42:45,689 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 3afdaaae >2014-04-29 02:42:45,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 63d0a3c3 >2014-04-29 02:42:45,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-45) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 63d0a3c3 >2014-04-29 02:42:45,933 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-45) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3afdaaae >2014-04-29 02:42:45,937 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-45) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-29 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-73) Start deleteAgedOutAuditLogs >2014-04-29 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-73) Finished deleteAgedOutAuditLogs >2014-04-29 03:36:04,967 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-84) Start refreshing all users data >2014-04-29 03:36:06,598 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VMs/Templates Ovf. >2014-04-29 03:36:06,599 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update VM OVFs in Data Center NFS >2014-04-29 03:36:06,600 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully updated VM OVFs in Data Center NFS >2014-04-29 03:36:06,600 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to update template OVFs in Data Center NFS >2014-04-29 03:36:06,601 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully updated templates OVFs in Data Center NFS >2014-04-29 03:36:06,601 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 03:36:06,602 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-41) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 03:42:45,942 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 2e392f6a >2014-04-29 03:42:46,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 56ce8213 >2014-04-29 03:42:46,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 56ce8213 >2014-04-29 03:42:46,288 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2e392f6a >2014-04-29 03:42:46,294 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-29 04:36:04,969 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-73) Start refreshing all users data >2014-04-29 04:36:06,602 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VMs/Templates Ovf. >2014-04-29 04:36:06,603 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update VM OVFs in Data Center NFS >2014-04-29 04:36:06,604 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated VM OVFs in Data Center NFS >2014-04-29 04:36:06,604 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to update template OVFs in Data Center NFS >2014-04-29 04:36:06,605 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully updated templates OVFs in Data Center NFS >2014-04-29 04:36:06,605 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 04:36:06,605 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-23) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 04:42:46,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-46) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 57c221b7 >2014-04-29 04:42:46,297 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-46) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: a33a264 >2014-04-29 04:42:46,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-46) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: a33a264 >2014-04-29 04:42:46,534 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-46) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 57c221b7 >2014-04-29 04:42:46,539 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-46) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-29 04:48:18,063 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space.gal domain has 1 GB of free space >2014-04-29 04:48:18,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space.storage domain has 1 GB of free space >2014-04-29 04:48:18,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space.iso domain has 1 GB of free space >2014-04-29 05:36:04,972 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-93) Start refreshing all users data >2014-04-29 05:36:06,605 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VMs/Templates Ovf. >2014-04-29 05:36:06,606 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update VM OVFs in Data Center NFS >2014-04-29 05:36:06,607 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated VM OVFs in Data Center NFS >2014-04-29 05:36:06,607 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to update template OVFs in Data Center NFS >2014-04-29 05:36:06,607 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully updated templates OVFs in Data Center NFS >2014-04-29 05:36:06,607 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 05:36:06,608 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-32) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 05:42:46,541 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-46) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 13c98ef1 >2014-04-29 05:42:46,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-46) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 15c77a5d >2014-04-29 05:42:46,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-46) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 15c77a5d >2014-04-29 05:42:46,783 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-46) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 13c98ef1 >2014-04-29 05:42:46,788 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-46) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-29 06:36:04,972 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) Start refreshing all users data >2014-04-29 06:36:06,608 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VMs/Templates Ovf. >2014-04-29 06:36:06,609 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update VM OVFs in Data Center NFS >2014-04-29 06:36:06,610 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated VM OVFs in Data Center NFS >2014-04-29 06:36:06,610 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to update template OVFs in Data Center NFS >2014-04-29 06:36:06,610 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully updated templates OVFs in Data Center NFS >2014-04-29 06:36:06,610 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 06:36:06,611 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-99) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 06:42:46,790 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 1a269407 >2014-04-29 06:42:46,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 5ac064fa >2014-04-29 06:42:47,022 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5ac064fa >2014-04-29 06:42:47,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 1a269407 >2014-04-29 06:42:47,028 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-29 07:36:04,974 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) Start refreshing all users data >2014-04-29 07:36:06,612 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VMs/Templates Ovf. >2014-04-29 07:36:06,612 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update VM OVFs in Data Center NFS >2014-04-29 07:36:06,613 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated VM OVFs in Data Center NFS >2014-04-29 07:36:06,613 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to update template OVFs in Data Center NFS >2014-04-29 07:36:06,614 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully updated templates OVFs in Data Center NFS >2014-04-29 07:36:06,614 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 07:36:06,614 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 07:42:47,031 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 5c1fc6db >2014-04-29 07:42:47,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 2a38831d >2014-04-29 07:42:47,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-50) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 2a38831d >2014-04-29 07:42:47,266 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-50) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 5c1fc6db >2014-04-29 07:42:47,271 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-50) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-29 08:02:40,391 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. gal domain has 0 GB of free space >2014-04-29 08:02:40,400 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. storage domain has 0 GB of free space >2014-04-29 08:02:40,407 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. iso domain has 0 GB of free space >2014-04-29 08:02:43,966 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space. Host 10.34.27.209 has less than 1000 MB of free space left on: /var/log/core, /tmp, /var/log, /var/run/vdsm/. >2014-04-29 08:04:06,037 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. Host 10.34.27.209 has less than 500 MB of free space left on: /var/log/core, /tmp, /var/log, /var/run/vdsm/. >2014-04-29 08:17:42,933 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. gal domain has 0 GB of free space >2014-04-29 08:17:42,939 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. storage domain has 0 GB of free space >2014-04-29 08:17:42,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. iso domain has 0 GB of free space >2014-04-29 08:19:12,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. Host 10.34.27.209 has less than 500 MB of free space left on: /var/log/core, /tmp, /var/log, /var/run/vdsm/. >2014-04-29 08:32:46,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. gal domain has 0 GB of free space >2014-04-29 08:32:46,617 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. storage domain has 0 GB of free space >2014-04-29 08:32:46,624 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. iso domain has 0 GB of free space >2014-04-29 08:34:17,885 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. Host 10.34.27.209 has less than 500 MB of free space left on: /var/log/core, /tmp, /var/log, /var/run/vdsm/. >2014-04-29 08:36:04,977 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-75) Start refreshing all users data >2014-04-29 08:36:06,615 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VMs/Templates Ovf. >2014-04-29 08:36:06,616 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update VM OVFs in Data Center NFS >2014-04-29 08:36:06,617 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated VM OVFs in Data Center NFS >2014-04-29 08:36:06,617 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to update template OVFs in Data Center NFS >2014-04-29 08:36:06,618 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully updated templates OVFs in Data Center NFS >2014-04-29 08:36:06,618 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 08:36:06,618 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-97) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 08:42:47,273 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) START, GetIsoListVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 6ae087e2 >2014-04-29 08:42:47,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) START, HsmGetIsoListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=f03e56af-56f8-4150-9517-271bec2305d9), log id: 3af93d3f >2014-04-29 08:42:47,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HsmGetIsoListVDSCommand] (pool-4-thread-48) FINISH, HsmGetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 3af93d3f >2014-04-29 08:42:47,510 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetIsoListVDSCommand] (pool-4-thread-48) FINISH, GetIsoListVDSCommand, return: [virtio-win-1.3.1.iso, virtio-win-1.3.3.iso, RHEV-toolsSetup_3.0_8.iso, en_windows_server_2008_r2_standard_enterprise_datacenter_and_web_x64_dvd_x15-59754.iso, RHEV-toolsSetup_3.1_9.iso, en_windows_xp_professional_with_service_pack_3_x86_cd_vl_x14-73974.iso, virtio-win-1.0.2-52454.iso, WANemv2.3.iso], log id: 6ae087e2 >2014-04-29 08:42:47,515 INFO [org.ovirt.engine.core.bll.IsoDomainListSyncronizer] (pool-4-thread-48) Finished automatic refresh process for ISO file type with success, for storage domain id 6ba7421b-4b80-4aff-ba01-1c08d3ef4619. >2014-04-29 08:47:47,751 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. gal domain has 0 GB of free space >2014-04-29 08:47:47,755 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. storage domain has 0 GB of free space >2014-04-29 08:47:47,762 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. iso domain has 0 GB of free space >2014-04-29 08:49:09,782 INFO [org.ovirt.engine.core.bll.LoginAdminUserCommand] (ajp-/127.0.0.1:8702-8) Running command: LoginAdminUserCommand internal: false. >2014-04-29 08:49:09,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-8) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-29 08:49:09,840 INFO [org.ovirt.engine.core.bll.LoginUserCommand] (ajp-/127.0.0.1:8702-10) Running command: LoginUserCommand internal: false. >2014-04-29 08:49:09,859 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-10) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: User admin@internal logged in. >2014-04-29 08:49:21,551 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-11) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 420e8f05 >2014-04-29 08:49:21,780 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-11) FINISH, GetVmsInfoVDSCommand, log id: 420e8f05 >2014-04-29 08:49:25,070 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-5) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 2026a7b5 >2014-04-29 08:49:25,296 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-5) FINISH, GetVmsInfoVDSCommand, log id: 2026a7b5 >2014-04-29 08:49:30,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Critical, Low disk space. Host 10.34.27.209 has less than 500 MB of free space left on: /var/log/core, /tmp, /var/log, /var/run/vdsm/. >2014-04-29 08:51:57,980 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [66679c28] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.27.209:/mnt/export value: STORAGE_CONNECTION >, sharedLocks= ] >2014-04-29 08:51:57,988 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [66679c28] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 08:51:57,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [66679c28] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.27.209:/mnt/export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4938d343 >2014-04-29 08:51:58,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-8) [66679c28] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 4938d343 >2014-04-29 08:51:58,543 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-8) [66679c28] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.27.209:/mnt/export value: STORAGE_CONNECTION >, sharedLocks= ] >2014-04-29 08:51:58,554 INFO [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp-/127.0.0.1:8702-7) START, GetExistingStorageDomainListQuery(refresh: true, filtered: false), log id: 1adf9b5b >2014-04-29 08:51:58,556 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-7) START, HSMGetStorageDomainsListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=00000000-0000-0000-0000-000000000000, storageType=NFS, storageDomainType=ImportExport, path=10.34.27.209:/mnt/export), log id: 140fca91 >2014-04-29 08:52:05,545 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-7) FINISH, HSMGetStorageDomainsListVDSCommand, return: [dd8be67a-986d-4465-977a-afbc5f37d283], log id: 140fca91 >2014-04-29 08:52:05,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp-/127.0.0.1:8702-7) START, HSMGetStorageDomainInfoVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storageDomainId=dd8be67a-986d-4465-977a-afbc5f37d283), log id: 126193d3 >2014-04-29 08:52:05,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp-/127.0.0.1:8702-7) FINISH, HSMGetStorageDomainInfoVDSCommand, return: org.ovirt.engine.core.common.utils.Pair@da319f53, log id: 126193d3 >2014-04-29 08:52:05,815 INFO [org.ovirt.engine.core.bll.storage.GetExistingStorageDomainListQuery] (ajp-/127.0.0.1:8702-7) FINISH, GetExistingStorageDomainListQuery, log id: 1adf9b5b >2014-04-29 08:52:05,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-4) [66ef54f2] START, HSMGetStorageDomainsListVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId=00000000-0000-0000-0000-000000000000, storageType=NFS, storageDomainType=ImportExport, path=), log id: 61e4c389 >2014-04-29 08:52:12,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainsListVDSCommand] (ajp-/127.0.0.1:8702-4) [66ef54f2] FINISH, HSMGetStorageDomainsListVDSCommand, return: [c0532ddb-5faf-4ab6-b93f-c490c588cae4, dd8be67a-986d-4465-977a-afbc5f37d283], log id: 61e4c389 >2014-04-29 08:52:12,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp-/127.0.0.1:8702-4) [66ef54f2] START, HSMGetStorageDomainInfoVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storageDomainId=dd8be67a-986d-4465-977a-afbc5f37d283), log id: 54abe53 >2014-04-29 08:52:12,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetStorageDomainInfoVDSCommand] (ajp-/127.0.0.1:8702-4) [66ef54f2] FINISH, HSMGetStorageDomainInfoVDSCommand, return: org.ovirt.engine.core.common.utils.Pair@da319f53, log id: 54abe53 >2014-04-29 08:52:12,664 INFO [org.ovirt.engine.core.bll.storage.AddExistingFileStorageDomainCommand] (ajp-/127.0.0.1:8702-4) [66ef54f2] Running command: AddExistingFileStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 08:52:12,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-4) [66ef54f2] START, GetStorageDomainStatsVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storageDomainId=dd8be67a-986d-4465-977a-afbc5f37d283), log id: 13d7f2cd >2014-04-29 08:52:12,895 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-4) [66ef54f2] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@a348c4c4, log id: 13d7f2cd >2014-04-29 08:52:12,904 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-4) [66ef54f2] Correlation ID: 66ef54f2, Job ID: f11119aa-0a45-477a-a6c3-d4324b29a82d, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export was added by admin@internal >2014-04-29 08:52:12,932 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-11) [43cb035a] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 08:52:12,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [43cb035a] START, DisconnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bdc5868c-9829-4c9d-80c4-a9328e54153c, connection: 10.34.27.209:/mnt/export, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1515264f >2014-04-29 08:52:21,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-11) [43cb035a] FINISH, DisconnectStorageServerVDSCommand, return: {bdc5868c-9829-4c9d-80c4-a9328e54153c=0}, log id: 1515264f >2014-04-29 08:52:44,988 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-10) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, vmIdList = null), log id: 703cc339 >2014-04-29 08:52:45,226 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-10) FINISH, GetVmsInfoVDSCommand, log id: 703cc339 >2014-04-29 08:52:57,932 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [6387506a] Lock Acquired to object EngineLock [exclusiveLocks= key: c0532ddb-5faf-4ab6-b93f-c490c588cae4 value: STORAGE >, sharedLocks= ] >2014-04-29 08:52:57,966 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-50) [6387506a] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: c0532ddb-5faf-4ab6-b93f-c490c588cae4 Type: Storage >2014-04-29 08:52:57,969 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-50) [6387506a] START, DeactivateStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 79de62ec >2014-04-29 08:52:58,250 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-50) [6387506a] Failed in DeactivateStorageDomainVDS method >2014-04-29 08:52:58,250 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-50) [6387506a] Error code StorageDomainActionError and error message IRSGenericException: IRSErrorException: Failed to DeactivateStorageDomainVDS, error = Error in storage domain action: ('sdUUID=c0532ddb-5faf-4ab6-b93f-c490c588cae4, spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=00000000-0000-0000-0000-000000000000, masterVersion=1',) >2014-04-29 08:52:58,250 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-50) [6387506a] IrsBroker::Failed::DeactivateStorageDomainVDS due to: IRSErrorException: IRSGenericException: IRSErrorException: Failed to DeactivateStorageDomainVDS, error = Error in storage domain action: ('sdUUID=c0532ddb-5faf-4ab6-b93f-c490c588cae4, spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=00000000-0000-0000-0000-000000000000, masterVersion=1',) >2014-04-29 08:52:58,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-50) [6387506a] START, SpmStopVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9), log id: 3a2a2b6a >2014-04-29 08:52:58,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-50) [6387506a] SpmStopVDSCommand::Stopping SPM on vds 10.34.1.251, pool id f03e56af-56f8-4150-9517-271bec2305d9 >2014-04-29 08:52:58,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-50) [6387506a] FINISH, SpmStopVDSCommand, log id: 3a2a2b6a >2014-04-29 08:52:58,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-50) [6387506a] Irs placed on server 56f84250-7f1e-46c8-924f-66c454fd877a failed. Proceed Failover >2014-04-29 08:52:58,999 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (pool-4-thread-50) [2de3dbfb] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:52:59,006 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) [2de3dbfb] Correlation ID: 2de3dbfb, Call Stack: null, Custom Event ID: -1, Message: Data Center is being initialized, please wait for initialization to complete. >2014-04-29 08:52:59,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-50) [2de3dbfb] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:52:59,406 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-50) [2de3dbfb] SPM Init: could not find reported vds or not up - pool:NFS vds_spm_id: 1 >2014-04-29 08:52:59,409 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-50) [2de3dbfb] SPM selection - vds seems as spm 10.34.1.251 >2014-04-29 08:52:59,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-50) [2de3dbfb] START, SpmStopVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9), log id: 20dea809 >2014-04-29 08:52:59,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (pool-4-thread-50) [2de3dbfb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:52:59,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (pool-4-thread-50) [2de3dbfb] HostName = 10.34.1.251 >2014-04-29 08:52:59,630 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (pool-4-thread-50) [2de3dbfb] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:52:59,630 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStopVDSCommand] (pool-4-thread-50) [2de3dbfb] FINISH, SpmStopVDSCommand, log id: 20dea809 >2014-04-29 08:52:59,630 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-50) [2de3dbfb] spm stop succeeded, continuing with spm selection >2014-04-29 08:52:59,632 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-50) [2de3dbfb] starting spm on vds 10.34.27.209, storage pool NFS, prevId 1, LVER 0 >2014-04-29 08:52:59,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-50) [2de3dbfb] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=true), log id: 17fead17 >2014-04-29 08:52:59,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-50) [2de3dbfb] spmStart polling started: taskId = 24dcac2e-2c27-4d16-8bef-69b43ef56c3a >2014-04-29 08:53:01,092 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (pool-4-thread-50) [2de3dbfb] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:53:01,093 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (pool-4-thread-50) [2de3dbfb] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:01,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-50) [2de3dbfb] spmStart polling ended: taskId = 24dcac2e-2c27-4d16-8bef-69b43ef56c3a task status = finished >2014-04-29 08:53:01,093 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-50) [2de3dbfb] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:01,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-50) [2de3dbfb] spmStart polling ended, spm status: Free >2014-04-29 08:53:01,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) [2de3dbfb] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=24dcac2e-2c27-4d16-8bef-69b43ef56c3a), log id: 1018145a >2014-04-29 08:53:01,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) [2de3dbfb] FINISH, HSMClearTaskVDSCommand, log id: 1018145a >2014-04-29 08:53:01,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (pool-4-thread-50) [2de3dbfb] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@13eb3b91, log id: 17fead17 >2014-04-29 08:53:01,537 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (pool-4-thread-50) [3893b6ce] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:53:01,543 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) [3893b6ce] Correlation ID: 3893b6ce, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:53:01,546 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-50) [3893b6ce] IrsBroker::Failed::DeactivateStorageDomainVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:53:01,550 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-50) [3893b6ce] FINISH, DeactivateStorageDomainVDSCommand, log id: 79de62ec >2014-04-29 08:53:01,551 ERROR [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-50) [3893b6ce] Command org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.irsbroker.IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed (Failed with error ENGINE and code 5001) >2014-04-29 08:53:01,561 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-50) [3893b6ce] Command [id=cc74a7ba-288c-4667-b042-e4184ebeb8bc]: Compensating CHANGED_STATUS_ONLY of org.ovirt.engine.core.common.businessentities.StoragePoolIsoMap; snapshot: EntityStatusSnapshot [id=storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, status=Unknown]. >2014-04-29 08:53:01,565 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) [3893b6ce] Correlation ID: 6387506a, Job ID: a5aae168-0ca2-4b0f-8280-016ccf05c193, Call Stack: null, Custom Event ID: -1, Message: Failed to deactivate Storage Domain gal (Data Center NFS). (User: admin@internal) >2014-04-29 08:53:01,568 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-50) [3893b6ce] Lock freed to object EngineLock [exclusiveLocks= key: c0532ddb-5faf-4ab6-b93f-c490c588cae4 value: STORAGE >, sharedLocks= ] >2014-04-29 08:53:02,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-2) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:53:02,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-2) HostName = 10.34.27.209 >2014-04-29 08:53:02,904 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-2) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:53:03,133 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:53:03,136 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:53:03,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 353234c9 >2014-04-29 08:53:03,361 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) spmStart polling started: taskId = 8c5e6cfe-1d9d-4c25-913b-7295f6e91067 >2014-04-29 08:53:04,583 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:53:04,583 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:04,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) spmStart polling ended: taskId = 8c5e6cfe-1d9d-4c25-913b-7295f6e91067 task status = finished >2014-04-29 08:53:04,583 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:04,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) spmStart polling ended, spm status: Free >2014-04-29 08:53:04,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-2) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=8c5e6cfe-1d9d-4c25-913b-7295f6e91067), log id: 73e4ba26 >2014-04-29 08:53:05,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-2) FINISH, HSMClearTaskVDSCommand, log id: 73e4ba26 >2014-04-29 08:53:05,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2a45e308, log id: 353234c9 >2014-04-29 08:53:05,028 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:53:05,035 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [523c37e6] Correlation ID: 523c37e6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:53:05,037 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:53:05,261 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:53:05,492 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:53:05,495 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 0 >2014-04-29 08:53:05,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7a694b0e >2014-04-29 08:53:05,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] spmStart polling started: taskId = 4a9ff8b2-ae3e-4710-9287-c8385d3768ec >2014-04-29 08:53:06,934 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:53:06,934 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:06,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] spmStart polling ended: taskId = 4a9ff8b2-ae3e-4710-9287-c8385d3768ec task status = finished >2014-04-29 08:53:06,934 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:07,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] spmStart polling ended, spm status: Free >2014-04-29 08:53:07,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=4a9ff8b2-ae3e-4710-9287-c8385d3768ec), log id: 7168edec >2014-04-29 08:53:07,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] FINISH, HSMClearTaskVDSCommand, log id: 7168edec >2014-04-29 08:53:07,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-2) [523c37e6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a3e0dae, log id: 7a694b0e >2014-04-29 08:53:07,399 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-2) [ff45100] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:53:07,407 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [ff45100] Correlation ID: ff45100, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:53:07,410 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-2) [ff45100] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:53:17,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:53:17,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-31) HostName = 10.34.1.251 >2014-04-29 08:53:17,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-31) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:53:18,098 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:53:18,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) SpmStatus on vds 56f84250-7f1e-46c8-924f-66c454fd877a: Free >2014-04-29 08:53:18,327 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) starting spm on vds 10.34.1.251, storage pool NFS, prevId 1, LVER 0 >2014-04-29 08:53:18,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=1, prevLVER=0, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: b783138 >2014-04-29 08:53:18,547 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling started: taskId = 4a9dd40c-53fd-453b-bb41-8897f43c808d >2014-04-29 08:53:19,778 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:53:19,779 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:19,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling ended: taskId = 4a9dd40c-53fd-453b-bb41-8897f43c808d task status = finished >2014-04-29 08:53:19,779 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:20,026 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) spmStart polling ended, spm status: Free >2014-04-29 08:53:20,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-31) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=4a9dd40c-53fd-453b-bb41-8897f43c808d), log id: 769e94a0 >2014-04-29 08:53:20,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, HSMClearTaskVDSCommand, log id: 769e94a0 >2014-04-29 08:53:20,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@200dcf5d, log id: b783138 >2014-04-29 08:53:20,251 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:53:20,258 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:53:20,480 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] Irs placed on server 56f84250-7f1e-46c8-924f-66c454fd877a failed. Proceed Failover >2014-04-29 08:53:20,707 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:53:20,710 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:53:20,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 31025b6c >2014-04-29 08:53:20,934 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] spmStart polling started: taskId = 79948ca9-3548-4b36-89e0-e9284eff1ce0 >2014-04-29 08:53:22,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:53:22,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:22,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] spmStart polling ended: taskId = 79948ca9-3548-4b36-89e0-e9284eff1ce0 task status = finished >2014-04-29 08:53:22,156 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:22,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] spmStart polling ended, spm status: Free >2014-04-29 08:53:22,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=79948ca9-3548-4b36-89e0-e9284eff1ce0), log id: 45afbcef >2014-04-29 08:53:22,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] FINISH, HSMClearTaskVDSCommand, log id: 45afbcef >2014-04-29 08:53:22,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-31) [6aa8f31b] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6a165f3, log id: 31025b6c >2014-04-29 08:53:22,599 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:53:22,608 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:53:33,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:53:33,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-79) HostName = 10.34.27.209 >2014-04-29 08:53:33,057 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-79) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:53:33,280 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:53:33,283 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:53:33,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 39b7653c >2014-04-29 08:53:33,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) spmStart polling started: taskId = 5e9b0091-68c0-4241-a5bf-a8d02fee74e8 >2014-04-29 08:53:34,733 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:53:34,733 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:34,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) spmStart polling ended: taskId = 5e9b0091-68c0-4241-a5bf-a8d02fee74e8 task status = finished >2014-04-29 08:53:34,733 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:34,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) spmStart polling ended, spm status: Free >2014-04-29 08:53:34,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-79) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=5e9b0091-68c0-4241-a5bf-a8d02fee74e8), log id: 7c3e641d >2014-04-29 08:53:35,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, HSMClearTaskVDSCommand, log id: 7c3e641d >2014-04-29 08:53:35,175 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@38902e89, log id: 39b7653c >2014-04-29 08:53:35,176 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:53:35,182 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:53:35,403 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:53:35,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:53:35,626 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:53:35,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1f8505eb >2014-04-29 08:53:35,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] spmStart polling started: taskId = 750713bd-dda9-4c39-bc2b-e6303298ad84 >2014-04-29 08:53:37,065 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:53:37,065 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:37,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] spmStart polling ended: taskId = 750713bd-dda9-4c39-bc2b-e6303298ad84 task status = finished >2014-04-29 08:53:37,065 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:37,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] spmStart polling ended, spm status: Free >2014-04-29 08:53:37,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=750713bd-dda9-4c39-bc2b-e6303298ad84), log id: 4d30bf90 >2014-04-29 08:53:37,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] FINISH, HSMClearTaskVDSCommand, log id: 4d30bf90 >2014-04-29 08:53:37,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-79) [1df4162] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@1fed6932, log id: 1f8505eb >2014-04-29 08:53:37,503 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-79) [1b9561fe] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:53:37,528 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-79) [1b9561fe] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:53:47,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:53:47,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-68) HostName = 10.34.1.251 >2014-04-29 08:53:47,968 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-68) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:53:48,187 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:53:48,190 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:53:48,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6c846eb4 >2014-04-29 08:53:48,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) spmStart polling started: taskId = 0bc5fb01-d631-4df6-834b-7bf9ffdee7f7 >2014-04-29 08:53:49,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:53:49,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:49,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) spmStart polling ended: taskId = 0bc5fb01-d631-4df6-834b-7bf9ffdee7f7 task status = finished >2014-04-29 08:53:49,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:49,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) spmStart polling ended, spm status: Free >2014-04-29 08:53:49,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=0bc5fb01-d631-4df6-834b-7bf9ffdee7f7), log id: da1cc4a >2014-04-29 08:53:50,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, HSMClearTaskVDSCommand, log id: da1cc4a >2014-04-29 08:53:50,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@69c9add7, log id: 6c846eb4 >2014-04-29 08:53:50,066 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:53:50,074 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:53:50,294 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] Irs placed on server 56f84250-7f1e-46c8-924f-66c454fd877a failed. Proceed Failover >2014-04-29 08:53:50,519 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:53:50,537 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:53:50,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5f9f7f33 >2014-04-29 08:53:50,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] spmStart polling started: taskId = 0ac35029-9d83-43a6-bd94-6933894dfe4b >2014-04-29 08:53:51,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:53:51,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:51,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] spmStart polling ended: taskId = 0ac35029-9d83-43a6-bd94-6933894dfe4b task status = finished >2014-04-29 08:53:51,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:53:52,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] spmStart polling ended, spm status: Free >2014-04-29 08:53:52,218 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=0ac35029-9d83-43a6-bd94-6933894dfe4b), log id: 6f767ce9 >2014-04-29 08:53:52,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] FINISH, HSMClearTaskVDSCommand, log id: 6f767ce9 >2014-04-29 08:53:52,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-68) [37b43483] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@2527c2f8, log id: 5f9f7f33 >2014-04-29 08:53:52,440 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-68) [310e0777] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:53:52,446 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-68) [310e0777] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:54:03,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:54:03,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-58) HostName = 10.34.27.209 >2014-04-29 08:54:03,069 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-58) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:54:03,297 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:54:03,301 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:54:03,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 34957cd6 >2014-04-29 08:54:03,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) spmStart polling started: taskId = 4194a4f1-40ba-4091-b0ba-dd95d9b2e9e3 >2014-04-29 08:54:04,747 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:54:04,747 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:04,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) spmStart polling ended: taskId = 4194a4f1-40ba-4091-b0ba-dd95d9b2e9e3 task status = finished >2014-04-29 08:54:04,747 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:04,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) spmStart polling ended, spm status: Free >2014-04-29 08:54:04,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=4194a4f1-40ba-4091-b0ba-dd95d9b2e9e3), log id: 17cefb95 >2014-04-29 08:54:05,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) FINISH, HSMClearTaskVDSCommand, log id: 17cefb95 >2014-04-29 08:54:05,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@297a6c6f, log id: 34957cd6 >2014-04-29 08:54:05,189 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-58) [246500af] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:54:05,196 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [246500af] Correlation ID: 246500af, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:54:05,198 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [246500af] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:54:05,419 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [246500af] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:54:05,642 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [246500af] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:54:05,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [246500af] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:54:05,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [246500af] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 7c3a5fd9 >2014-04-29 08:54:05,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [246500af] spmStart polling started: taskId = f1d3ec5e-42d5-4236-896c-fa81be8ca461 >2014-04-29 08:54:07,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [246500af] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:54:07,088 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [246500af] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:07,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [246500af] spmStart polling ended: taskId = f1d3ec5e-42d5-4236-896c-fa81be8ca461 task status = finished >2014-04-29 08:54:07,088 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [246500af] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:07,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [246500af] spmStart polling ended, spm status: Free >2014-04-29 08:54:07,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) [246500af] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=f1d3ec5e-42d5-4236-896c-fa81be8ca461), log id: 67152205 >2014-04-29 08:54:07,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-58) [246500af] FINISH, HSMClearTaskVDSCommand, log id: 67152205 >2014-04-29 08:54:07,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-58) [246500af] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@45e031ea, log id: 7c3a5fd9 >2014-04-29 08:54:07,537 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-58) [1ed1d7d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:54:07,543 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [1ed1d7d] Correlation ID: 1ed1d7d, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:54:07,545 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-58) [1ed1d7d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:54:17,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:54:17,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) HostName = 10.34.1.251 >2014-04-29 08:54:17,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-48) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:54:18,206 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:54:18,210 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:54:18,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 61d3bd86 >2014-04-29 08:54:18,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) spmStart polling started: taskId = a434bb5c-eba7-402a-a7c6-0855407dc4b3 >2014-04-29 08:54:19,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:54:19,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:19,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) spmStart polling ended: taskId = a434bb5c-eba7-402a-a7c6-0855407dc4b3 task status = finished >2014-04-29 08:54:19,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:19,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) spmStart polling ended, spm status: Free >2014-04-29 08:54:19,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-48) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=a434bb5c-eba7-402a-a7c6-0855407dc4b3), log id: 10d5fa7 >2014-04-29 08:54:20,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-48) FINISH, HSMClearTaskVDSCommand, log id: 10d5fa7 >2014-04-29 08:54:20,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@4ff4052, log id: 61d3bd86 >2014-04-29 08:54:20,141 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:54:20,148 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:54:20,370 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:54:20,591 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:54:20,595 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:54:20,595 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 54d4f77d >2014-04-29 08:54:20,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] spmStart polling started: taskId = 96d22070-962f-431b-a8d9-5741d979048b >2014-04-29 08:54:22,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:54:22,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:22,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] spmStart polling ended: taskId = 96d22070-962f-431b-a8d9-5741d979048b task status = finished >2014-04-29 08:54:22,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:22,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] spmStart polling ended, spm status: Free >2014-04-29 08:54:22,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=96d22070-962f-431b-a8d9-5741d979048b), log id: 5b44f9ed >2014-04-29 08:54:22,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] FINISH, HSMClearTaskVDSCommand, log id: 5b44f9ed >2014-04-29 08:54:22,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-48) [79f9396c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7be19cfc, log id: 54d4f77d >2014-04-29 08:54:22,475 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-48) [1909a3ee] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:54:22,482 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-48) [1909a3ee] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:54:32,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:54:32,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-73) HostName = 10.34.1.251 >2014-04-29 08:54:32,923 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-73) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:54:33,142 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:54:33,145 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:54:33,146 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 27b42666 >2014-04-29 08:54:33,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) spmStart polling started: taskId = a2ac66b6-d856-4e13-a883-9da43e3c12bc >2014-04-29 08:54:34,584 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:54:34,584 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:34,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) spmStart polling ended: taskId = a2ac66b6-d856-4e13-a883-9da43e3c12bc task status = finished >2014-04-29 08:54:34,584 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:34,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) spmStart polling ended, spm status: Free >2014-04-29 08:54:34,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-73) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=a2ac66b6-d856-4e13-a883-9da43e3c12bc), log id: 772c17d4 >2014-04-29 08:54:35,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, HSMClearTaskVDSCommand, log id: 772c17d4 >2014-04-29 08:54:35,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@35e66341, log id: 27b42666 >2014-04-29 08:54:35,020 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:54:35,026 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:54:35,244 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] Irs placed on server 56f84250-7f1e-46c8-924f-66c454fd877a failed. Proceed Failover >2014-04-29 08:54:35,468 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:54:35,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:54:35,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6b15f66 >2014-04-29 08:54:35,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] spmStart polling started: taskId = 9ca07c6e-528d-4ed5-9e62-08a1fc112064 >2014-04-29 08:54:36,916 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:54:36,916 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:36,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] spmStart polling ended: taskId = 9ca07c6e-528d-4ed5-9e62-08a1fc112064 task status = finished >2014-04-29 08:54:36,917 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:37,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] spmStart polling ended, spm status: Free >2014-04-29 08:54:37,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=9ca07c6e-528d-4ed5-9e62-08a1fc112064), log id: 4fc92d60 >2014-04-29 08:54:37,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] FINISH, HSMClearTaskVDSCommand, log id: 4fc92d60 >2014-04-29 08:54:37,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-73) [287e5c0c] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7e99f45e, log id: 6b15f66 >2014-04-29 08:54:37,360 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-73) [2d7a8841] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:54:37,367 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-73) [2d7a8841] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:54:47,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:54:47,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-75) HostName = 10.34.27.209 >2014-04-29 08:54:47,811 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-75) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:54:48,055 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:54:48,057 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:54:48,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6bd00299 >2014-04-29 08:54:48,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) spmStart polling started: taskId = 990b314f-c58c-4d7d-a3eb-65c8479f63b9 >2014-04-29 08:54:49,624 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:54:49,624 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:49,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) spmStart polling ended: taskId = 990b314f-c58c-4d7d-a3eb-65c8479f63b9 task status = finished >2014-04-29 08:54:49,624 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:49,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) spmStart polling ended, spm status: Free >2014-04-29 08:54:49,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-75) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=990b314f-c58c-4d7d-a3eb-65c8479f63b9), log id: 73f755c3 >2014-04-29 08:54:50,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, HSMClearTaskVDSCommand, log id: 73f755c3 >2014-04-29 08:54:50,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5e707ae2, log id: 6bd00299 >2014-04-29 08:54:50,067 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:54:50,074 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:54:50,299 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:54:50,526 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:54:50,529 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:54:50,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1be6b3c >2014-04-29 08:54:50,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] spmStart polling started: taskId = b67072d6-a3d6-4547-ab43-f7dffc49dd31 >2014-04-29 08:54:51,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:54:51,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:51,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] spmStart polling ended: taskId = b67072d6-a3d6-4547-ab43-f7dffc49dd31 task status = finished >2014-04-29 08:54:51,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:54:52,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] spmStart polling ended, spm status: Free >2014-04-29 08:54:52,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=b67072d6-a3d6-4547-ab43-f7dffc49dd31), log id: 417f91a0 >2014-04-29 08:54:52,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] FINISH, HSMClearTaskVDSCommand, log id: 417f91a0 >2014-04-29 08:54:52,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-75) [2a93bbde] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@debd6da, log id: 1be6b3c >2014-04-29 08:54:52,404 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-75) [76668959] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:54:52,411 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-75) [76668959] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:55:02,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:55:02,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-55) HostName = 10.34.1.251 >2014-04-29 08:55:02,863 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-55) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:55:03,086 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:55:03,089 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:55:03,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6a52382c >2014-04-29 08:55:03,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling started: taskId = 98468c4f-1bcb-478b-a571-abe0e54f85a1 >2014-04-29 08:55:04,720 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:55:04,721 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:04,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling ended: taskId = 98468c4f-1bcb-478b-a571-abe0e54f85a1 task status = finished >2014-04-29 08:55:04,721 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:04,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) spmStart polling ended, spm status: Free >2014-04-29 08:55:04,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=98468c4f-1bcb-478b-a571-abe0e54f85a1), log id: 653f35b9 >2014-04-29 08:55:05,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, HSMClearTaskVDSCommand, log id: 653f35b9 >2014-04-29 08:55:05,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3207afc, log id: 6a52382c >2014-04-29 08:55:05,167 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:55:05,179 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:55:05,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:55:05,624 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:55:05,627 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:55:05,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: e9a7777 >2014-04-29 08:55:05,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] spmStart polling started: taskId = 6d2d65c9-9512-4a66-a8d0-1e4e8d927f34 >2014-04-29 08:55:07,070 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:55:07,071 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:07,071 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] spmStart polling ended: taskId = 6d2d65c9-9512-4a66-a8d0-1e4e8d927f34 task status = finished >2014-04-29 08:55:07,071 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:07,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] spmStart polling ended, spm status: Free >2014-04-29 08:55:07,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=6d2d65c9-9512-4a66-a8d0-1e4e8d927f34), log id: 651da6bc >2014-04-29 08:55:07,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] FINISH, HSMClearTaskVDSCommand, log id: 651da6bc >2014-04-29 08:55:07,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [496a37fe] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@23fa9fef, log id: e9a7777 >2014-04-29 08:55:07,512 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:55:07,533 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:55:17,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:55:18,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) HostName = 10.34.1.251 >2014-04-29 08:55:18,000 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:55:18,225 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:55:18,228 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:55:18,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5f861441 >2014-04-29 08:55:18,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) spmStart polling started: taskId = a5257a9a-00c9-48c8-9ada-d92dd1578cdf >2014-04-29 08:55:19,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:55:19,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:19,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) spmStart polling ended: taskId = a5257a9a-00c9-48c8-9ada-d92dd1578cdf task status = finished >2014-04-29 08:55:19,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:19,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) spmStart polling ended, spm status: Free >2014-04-29 08:55:19,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-76) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=a5257a9a-00c9-48c8-9ada-d92dd1578cdf), log id: 11adfcfc >2014-04-29 08:55:20,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, HSMClearTaskVDSCommand, log id: 11adfcfc >2014-04-29 08:55:20,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@41aed3d7, log id: 5f861441 >2014-04-29 08:55:20,127 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:55:20,133 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] Correlation ID: 3cd5e9f1, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:55:20,135 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:55:20,357 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:55:20,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:55:20,583 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:55:20,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 29a4d85f >2014-04-29 08:55:20,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] spmStart polling started: taskId = ffa66f41-4a03-47f4-95b1-461ec4441fe4 >2014-04-29 08:55:22,030 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:55:22,030 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:22,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] spmStart polling ended: taskId = ffa66f41-4a03-47f4-95b1-461ec4441fe4 task status = finished >2014-04-29 08:55:22,030 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:22,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] spmStart polling ended, spm status: Free >2014-04-29 08:55:22,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=ffa66f41-4a03-47f4-95b1-461ec4441fe4), log id: 5a332896 >2014-04-29 08:55:22,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] FINISH, HSMClearTaskVDSCommand, log id: 5a332896 >2014-04-29 08:55:22,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [3cd5e9f1] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@43f4aaec, log id: 29a4d85f >2014-04-29 08:55:22,467 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:55:22,473 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [74018ece] Correlation ID: 74018ece, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:55:22,478 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:55:33,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:55:33,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-30) HostName = 10.34.1.251 >2014-04-29 08:55:33,101 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-30) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:55:33,320 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-30) hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:55:33,324 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-30) starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:55:33,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 73cf3674 >2014-04-29 08:55:33,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) spmStart polling started: taskId = 1efc51fa-f2d0-44c0-8825-74f22b518531 >2014-04-29 08:55:34,761 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:55:34,762 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:34,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) spmStart polling ended: taskId = 1efc51fa-f2d0-44c0-8825-74f22b518531 task status = finished >2014-04-29 08:55:34,762 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:34,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) spmStart polling ended, spm status: Free >2014-04-29 08:55:34,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-30) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=1efc51fa-f2d0-44c0-8825-74f22b518531), log id: 1a1d771d >2014-04-29 08:55:35,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-30) FINISH, HSMClearTaskVDSCommand, log id: 1a1d771d >2014-04-29 08:55:35,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@6e05e89d, log id: 73cf3674 >2014-04-29 08:55:35,208 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:55:35,214 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:55:35,432 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] Irs placed on server 56f84250-7f1e-46c8-924f-66c454fd877a failed. Proceed Failover >2014-04-29 08:55:35,835 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:55:35,837 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:55:35,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 68d96e2c >2014-04-29 08:55:36,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] spmStart polling started: taskId = 92100257-3984-4f01-a5c9-68aa0a5dc5af >2014-04-29 08:55:37,281 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:55:37,281 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:37,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] spmStart polling ended: taskId = 92100257-3984-4f01-a5c9-68aa0a5dc5af task status = finished >2014-04-29 08:55:37,281 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:37,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] spmStart polling ended, spm status: Free >2014-04-29 08:55:37,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=92100257-3984-4f01-a5c9-68aa0a5dc5af), log id: dacb7e1 >2014-04-29 08:55:37,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] FINISH, HSMClearTaskVDSCommand, log id: dacb7e1 >2014-04-29 08:55:37,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-30) [45ead3d9] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@f4f4df9, log id: 68d96e2c >2014-04-29 08:55:37,726 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-30) [1f881e49] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:55:37,732 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-30) [1f881e49] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:55:48,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:55:48,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] HostName = 10.34.27.209 >2014-04-29 08:55:48,179 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:55:48,402 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:55:48,405 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:55:48,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 437fe297 >2014-04-29 08:55:48,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] spmStart polling started: taskId = 65b7da17-04d8-4cc9-8299-06dca6bc9922 >2014-04-29 08:55:49,853 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:55:49,853 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:49,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] spmStart polling ended: taskId = 65b7da17-04d8-4cc9-8299-06dca6bc9922 task status = finished >2014-04-29 08:55:49,853 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:50,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] spmStart polling ended, spm status: Free >2014-04-29 08:55:50,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=65b7da17-04d8-4cc9-8299-06dca6bc9922), log id: 5e25a87e >2014-04-29 08:55:50,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] FINISH, HSMClearTaskVDSCommand, log id: 5e25a87e >2014-04-29 08:55:50,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [2d1d982] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@42ad4bf5, log id: 437fe297 >2014-04-29 08:55:50,302 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:55:50,311 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:55:50,533 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:55:50,754 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:55:50,757 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:55:50,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 19a7033d >2014-04-29 08:55:50,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] spmStart polling started: taskId = a790d2d5-d28f-4579-a944-a7f5f4c1f51d >2014-04-29 08:55:52,204 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:55:52,204 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:52,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] spmStart polling ended: taskId = a790d2d5-d28f-4579-a944-a7f5f4c1f51d task status = finished >2014-04-29 08:55:52,206 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:55:52,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] spmStart polling ended, spm status: Free >2014-04-29 08:55:52,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=a790d2d5-d28f-4579-a944-a7f5f4c1f51d), log id: 319af44 >2014-04-29 08:55:52,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] FINISH, HSMClearTaskVDSCommand, log id: 319af44 >2014-04-29 08:55:52,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-11) [7f55e405] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@aedf730, log id: 19a7033d >2014-04-29 08:55:52,650 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-11) [cff75eb] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:55:52,676 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-11) [cff75eb] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:56:03,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:56:03,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) HostName = 10.34.1.251 >2014-04-29 08:56:03,295 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-7) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:56:03,522 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:56:03,525 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:56:03,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 29f89e81 >2014-04-29 08:56:03,748 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) spmStart polling started: taskId = 03ace39f-5100-4e13-955c-b09a75df72f1 >2014-04-29 08:56:04,980 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:56:04,980 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:04,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) spmStart polling ended: taskId = 03ace39f-5100-4e13-955c-b09a75df72f1 task status = finished >2014-04-29 08:56:04,980 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:05,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) spmStart polling ended, spm status: Free >2014-04-29 08:56:05,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-7) START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=03ace39f-5100-4e13-955c-b09a75df72f1), log id: 50b00e70 >2014-04-29 08:56:05,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, HSMClearTaskVDSCommand, log id: 50b00e70 >2014-04-29 08:56:05,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7dc8be4e, log id: 29f89e81 >2014-04-29 08:56:05,423 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-7) [6463366] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:56:05,431 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [6463366] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:56:05,655 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [6463366] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:56:05,878 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [6463366] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:56:05,882 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [6463366] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:56:05,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [6463366] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 45986104 >2014-04-29 08:56:06,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [6463366] spmStart polling started: taskId = e37cb15b-ebf7-4684-aae9-75c3b0b2abd2 >2014-04-29 08:56:07,330 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [6463366] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:56:07,330 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [6463366] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:07,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [6463366] spmStart polling ended: taskId = e37cb15b-ebf7-4684-aae9-75c3b0b2abd2 task status = finished >2014-04-29 08:56:07,330 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [6463366] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:07,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [6463366] spmStart polling ended, spm status: Free >2014-04-29 08:56:07,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-7) [6463366] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=e37cb15b-ebf7-4684-aae9-75c3b0b2abd2), log id: 1efc0049 >2014-04-29 08:56:07,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-7) [6463366] FINISH, HSMClearTaskVDSCommand, log id: 1efc0049 >2014-04-29 08:56:07,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-7) [6463366] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@312e7b65, log id: 45986104 >2014-04-29 08:56:07,777 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-7) [62a73d1c] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:56:07,785 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-7) [62a73d1c] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:56:18,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:56:18,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] HostName = 10.34.1.251 >2014-04-29 08:56:18,247 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:56:18,471 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:56:18,476 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:56:18,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 2cc4ef09 >2014-04-29 08:56:18,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] spmStart polling started: taskId = 4eba976d-7b29-4ddf-af58-b8319432b04a >2014-04-29 08:56:19,920 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:56:19,920 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:19,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] spmStart polling ended: taskId = 4eba976d-7b29-4ddf-af58-b8319432b04a task status = finished >2014-04-29 08:56:19,920 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:20,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] spmStart polling ended, spm status: Free >2014-04-29 08:56:20,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=4eba976d-7b29-4ddf-af58-b8319432b04a), log id: 100ac9ef >2014-04-29 08:56:20,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] FINISH, HSMClearTaskVDSCommand, log id: 100ac9ef >2014-04-29 08:56:20,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1b1a165d] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@3e4aa047, log id: 2cc4ef09 >2014-04-29 08:56:20,391 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:56:20,399 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [1560b544] Correlation ID: 1560b544, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:56:20,402 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:56:20,623 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:56:20,845 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:56:20,848 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:56:20,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 119529c9 >2014-04-29 08:56:21,072 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] spmStart polling started: taskId = 080ed9f6-1758-41bd-b11b-8a77e6b8e801 >2014-04-29 08:56:22,308 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:56:22,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:22,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] spmStart polling ended: taskId = 080ed9f6-1758-41bd-b11b-8a77e6b8e801 task status = finished >2014-04-29 08:56:22,309 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:22,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] spmStart polling ended, spm status: Free >2014-04-29 08:56:22,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=080ed9f6-1758-41bd-b11b-8a77e6b8e801), log id: 11bc6663 >2014-04-29 08:56:22,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] FINISH, HSMClearTaskVDSCommand, log id: 11bc6663 >2014-04-29 08:56:22,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-55) [1560b544] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@17d6e4bb, log id: 119529c9 >2014-04-29 08:56:22,757 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-55) [1d42bd37] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:56:22,765 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [1d42bd37] Correlation ID: 1d42bd37, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:56:22,768 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-55) [1d42bd37] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:56:33,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:56:33,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] HostName = 10.34.1.251 >2014-04-29 08:56:33,426 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:56:33,648 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:56:33,651 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:56:33,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 5a9a40d8 >2014-04-29 08:56:33,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] spmStart polling started: taskId = be5319e8-ac16-4717-8050-e77dd3ac16bd >2014-04-29 08:56:35,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:56:35,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:35,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] spmStart polling ended: taskId = be5319e8-ac16-4717-8050-e77dd3ac16bd task status = finished >2014-04-29 08:56:35,102 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:35,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] spmStart polling ended, spm status: Free >2014-04-29 08:56:35,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=be5319e8-ac16-4717-8050-e77dd3ac16bd), log id: 7fc16760 >2014-04-29 08:56:35,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] FINISH, HSMClearTaskVDSCommand, log id: 7fc16760 >2014-04-29 08:56:35,562 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [74018ece] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@180bcc38, log id: 5a9a40d8 >2014-04-29 08:56:35,564 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:56:35,572 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:56:35,801 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] Irs placed on server 53fd133a-9dc6-11e2-abd9-001a4a223991 failed. Proceed Failover >2014-04-29 08:56:36,023 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:56:36,026 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:56:36,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 1736e9c1 >2014-04-29 08:56:36,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] spmStart polling started: taskId = 9fd2a35d-b6e3-4a6b-abae-1eb7ff3bc61c >2014-04-29 08:56:38,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:56:38,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:38,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] spmStart polling ended: taskId = 9fd2a35d-b6e3-4a6b-abae-1eb7ff3bc61c task status = finished >2014-04-29 08:56:38,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:38,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] spmStart polling ended, spm status: Free >2014-04-29 08:56:38,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=9fd2a35d-b6e3-4a6b-abae-1eb7ff3bc61c), log id: 36c3afc3 >2014-04-29 08:56:39,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] FINISH, HSMClearTaskVDSCommand, log id: 36c3afc3 >2014-04-29 08:56:39,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-76) [2e8c7cdd] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@34551b8d, log id: 1736e9c1 >2014-04-29 08:56:39,144 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-76) [3f83e0d2] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:56:39,151 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-76) [3f83e0d2] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:56:49,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:56:49,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) HostName = 10.34.1.251 >2014-04-29 08:56:49,666 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-27) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:56:49,893 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:56:49,895 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:56:49,896 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 38b64d7c >2014-04-29 08:56:50,123 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling started: taskId = a9f2f1cd-cac0-40ad-8ac5-c4277761147c >2014-04-29 08:56:51,352 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:56:51,353 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:51,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling ended: taskId = a9f2f1cd-cac0-40ad-8ac5-c4277761147c task status = finished >2014-04-29 08:56:51,353 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:51,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) spmStart polling ended, spm status: Free >2014-04-29 08:56:51,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=a9f2f1cd-cac0-40ad-8ac5-c4277761147c), log id: 7c7853f9 >2014-04-29 08:56:51,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, HSMClearTaskVDSCommand, log id: 7c7853f9 >2014-04-29 08:56:51,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@11d106e2, log id: 38b64d7c >2014-04-29 08:56:51,794 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:56:51,800 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:56:52,019 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] Irs placed on server 56f84250-7f1e-46c8-924f-66c454fd877a failed. Proceed Failover >2014-04-29 08:56:52,249 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:56:52,252 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:56:52,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 63681629 >2014-04-29 08:56:52,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] spmStart polling started: taskId = 99aa8d19-a95d-454f-9c61-3014dbeacfd3 >2014-04-29 08:56:53,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:56:53,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:53,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] spmStart polling ended: taskId = 99aa8d19-a95d-454f-9c61-3014dbeacfd3 task status = finished >2014-04-29 08:56:53,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:56:53,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] spmStart polling ended, spm status: Free >2014-04-29 08:56:53,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=99aa8d19-a95d-454f-9c61-3014dbeacfd3), log id: 6435b0e6 >2014-04-29 08:56:54,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] FINISH, HSMClearTaskVDSCommand, log id: 6435b0e6 >2014-04-29 08:56:54,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-27) [321ae891] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@7186393d, log id: 63681629 >2014-04-29 08:56:54,143 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-27) [23180e05] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:56:54,150 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-27) [23180e05] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:57:04,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:57:04,600 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) HostName = 10.34.27.209 >2014-04-29 08:57:04,600 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-64) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:57:05,000 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:57:05,003 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:57:05,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6e877976 >2014-04-29 08:57:05,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling started: taskId = dab26a2d-fe7d-4522-be20-ddf2ad49342b >2014-04-29 08:57:07,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:57:07,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:07,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling ended: taskId = dab26a2d-fe7d-4522-be20-ddf2ad49342b task status = finished >2014-04-29 08:57:07,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:07,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) spmStart polling ended, spm status: Free >2014-04-29 08:57:07,907 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-64) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=dab26a2d-fe7d-4522-be20-ddf2ad49342b), log id: 2586d711 >2014-04-29 08:57:08,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, HSMClearTaskVDSCommand, log id: 2586d711 >2014-04-29 08:57:08,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@746f427a, log id: 6e877976 >2014-04-29 08:57:08,128 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:57:08,136 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:57:08,359 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] Irs placed on server 56f84250-7f1e-46c8-924f-66c454fd877a failed. Proceed Failover >2014-04-29 08:57:08,581 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:57:08,585 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:57:08,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: d78a360 >2014-04-29 08:57:08,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] spmStart polling started: taskId = 5d916a9e-0a05-46a9-bb07-3edf340d9764 >2014-04-29 08:57:10,030 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:57:10,030 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:10,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] spmStart polling ended: taskId = 5d916a9e-0a05-46a9-bb07-3edf340d9764 task status = finished >2014-04-29 08:57:10,031 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:10,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] spmStart polling ended, spm status: Free >2014-04-29 08:57:10,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=5d916a9e-0a05-46a9-bb07-3edf340d9764), log id: 7a5fbd57 >2014-04-29 08:57:10,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] FINISH, HSMClearTaskVDSCommand, log id: 7a5fbd57 >2014-04-29 08:57:10,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-64) [3475934e] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5d389d14, log id: d78a360 >2014-04-29 08:57:10,473 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-64) [2fd0bbdc] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:57:10,481 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-64) [2fd0bbdc] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:57:20,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:57:20,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-52) HostName = 10.34.27.209 >2014-04-29 08:57:20,931 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-52) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:57:21,155 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:57:21,157 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:57:21,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 41d46fb5 >2014-04-29 08:57:21,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) spmStart polling started: taskId = 5acee10b-683a-461e-97a2-d03d2ee6255c >2014-04-29 08:57:22,602 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:57:22,603 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:22,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) spmStart polling ended: taskId = 5acee10b-683a-461e-97a2-d03d2ee6255c task status = finished >2014-04-29 08:57:22,603 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:22,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) spmStart polling ended, spm status: Free >2014-04-29 08:57:22,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-52) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=5acee10b-683a-461e-97a2-d03d2ee6255c), log id: 4ab61571 >2014-04-29 08:57:23,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, HSMClearTaskVDSCommand, log id: 4ab61571 >2014-04-29 08:57:23,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@53c62b29, log id: 41d46fb5 >2014-04-29 08:57:23,041 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:57:23,048 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [773bbbb6] Correlation ID: 773bbbb6, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:57:23,050 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:57:23,275 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] Irs placed on server 56f84250-7f1e-46c8-924f-66c454fd877a failed. Proceed Failover >2014-04-29 08:57:23,498 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:57:23,501 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:57:23,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 23a4960a >2014-04-29 08:57:23,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] spmStart polling started: taskId = 3d1bf6e6-ba1c-4204-b9ba-96ccd942e4bb >2014-04-29 08:57:24,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:57:24,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:24,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] spmStart polling ended: taskId = 3d1bf6e6-ba1c-4204-b9ba-96ccd942e4bb task status = finished >2014-04-29 08:57:24,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:25,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] spmStart polling ended, spm status: Free >2014-04-29 08:57:25,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=3d1bf6e6-ba1c-4204-b9ba-96ccd942e4bb), log id: 55975141 >2014-04-29 08:57:25,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] FINISH, HSMClearTaskVDSCommand, log id: 55975141 >2014-04-29 08:57:25,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-52) [773bbbb6] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@306cced8, log id: 23a4960a >2014-04-29 08:57:25,388 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-52) [377fdfbe] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:57:25,395 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [377fdfbe] Correlation ID: 377fdfbe, Call Stack: null, Custom Event ID: -1, Message: Invalid status on Data Center NFS. Setting status to Non Responsive. >2014-04-29 08:57:25,398 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-52) [377fdfbe] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:57:35,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:57:35,847 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) HostName = 10.34.27.209 >2014-04-29 08:57:35,847 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-19) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:57:36,117 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:57:36,120 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:57:36,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 15b63b2 >2014-04-29 08:57:36,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) spmStart polling started: taskId = 21d25d35-b08b-42f9-b4cf-a23d954b43d8 >2014-04-29 08:57:38,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) Failed in HSMGetTaskStatusVDS method >2014-04-29 08:57:38,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:38,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) spmStart polling ended: taskId = 21d25d35-b08b-42f9-b4cf-a23d954b43d8 task status = finished >2014-04-29 08:57:38,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:39,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) spmStart polling ended, spm status: Free >2014-04-29 08:57:39,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-19) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=21d25d35-b08b-42f9-b4cf-a23d954b43d8), log id: 151ecfe >2014-04-29 08:57:39,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, HSMClearTaskVDSCommand, log id: 151ecfe >2014-04-29 08:57:39,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@69b11f30, log id: 15b63b2 >2014-04-29 08:57:39,221 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:57:39,234 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:57:39,457 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] Irs placed on server 56f84250-7f1e-46c8-924f-66c454fd877a failed. Proceed Failover >2014-04-29 08:57:39,683 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] hostFromVds::selectedVds - 10.34.27.209, spmStatus Free, storage pool NFS >2014-04-29 08:57:39,686 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] starting spm on vds 10.34.27.209, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:57:39,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] START, SpmStartVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 255c24cb >2014-04-29 08:57:39,910 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] spmStart polling started: taskId = 53c03041-342f-4806-af43-979a44901b93 >2014-04-29 08:57:41,140 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] Failed in HSMGetTaskStatusVDS method >2014-04-29 08:57:41,140 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetTaskStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] Error code GeneralException and error message VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:41,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] spmStart polling ended: taskId = 53c03041-342f-4806-af43-979a44901b93 task status = finished >2014-04-29 08:57:41,140 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] Start SPM Task failed - result: cleanSuccess, message: VDSGenericException: VDSErrorException: Failed to HSMGetTaskStatusVDS, error = [Errno 28] No space left on device >2014-04-29 08:57:41,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] spmStart polling ended, spm status: Free >2014-04-29 08:57:41,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] START, HSMClearTaskVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, taskId=53c03041-342f-4806-af43-979a44901b93), log id: 3cc08ef3 >2014-04-29 08:57:41,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] FINISH, HSMClearTaskVDSCommand, log id: 3cc08ef3 >2014-04-29 08:57:41,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-19) [6b76be85] FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@39f706b4, log id: 255c24cb >2014-04-29 08:57:41,588 INFO [org.ovirt.engine.core.bll.storage.SetStoragePoolStatusCommand] (DefaultQuartzScheduler_Worker-19) [8d93127] Running command: SetStoragePoolStatusCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 08:57:41,600 ERROR [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-19) [8d93127] IrsBroker::Failed::GetStoragePoolInfoVDS due to: IrsSpmStartFailedException: IRSGenericException: IRSErrorException: SpmStart failed >2014-04-29 08:57:52,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand return value > >TaskStatusListReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=654, mMessage=Not SPM]] > >2014-04-29 08:57:52,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-51) HostName = 10.34.27.209 >2014-04-29 08:57:52,053 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksStatusesVDSCommand] (DefaultQuartzScheduler_Worker-51) Command HSMGetAllTasksStatusesVDS execution failed. Exception: IRSNonOperationalException: IRSGenericException: IRSErrorException: IRSNonOperationalException: Not SPM >2014-04-29 08:57:54,258 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-51) hostFromVds::selectedVds - 10.34.1.251, spmStatus Free, storage pool NFS >2014-04-29 08:57:54,260 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-51) starting spm on vds 10.34.1.251, storage pool NFS, prevId -1, LVER 1 >2014-04-29 08:57:54,261 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) START, SpmStartVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, prevId=-1, prevLVER=1, storagePoolFormatType=V3, recoveryMode=Manual, SCSIFencing=false), log id: 6950052d >2014-04-29 08:57:54,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) spmStart polling started: taskId = 327938ff-fd9a-42ff-baef-c3d00aa3a39a >2014-04-29 08:57:55,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) spmStart polling ended: taskId = 327938ff-fd9a-42ff-baef-c3d00aa3a39a task status = finished >2014-04-29 08:57:55,916 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) spmStart polling ended, spm status: SPM >2014-04-29 08:57:55,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=327938ff-fd9a-42ff-baef-c3d00aa3a39a), log id: 270cfe33 >2014-04-29 08:57:56,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, HSMClearTaskVDSCommand, log id: 270cfe33 >2014-04-29 08:57:56,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.SpmStartVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, SpmStartVDSCommand, return: org.ovirt.engine.core.common.businessentities.SpmStatusResult@5e0437a2, log id: 6950052d >2014-04-29 08:57:56,135 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (DefaultQuartzScheduler_Worker-51) Initialize Irs proxy from vds: 10.34.1.251 >2014-04-29 08:57:56,139 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Storage Pool Manager runs on Host 10.34.1.251 (Address: 10.34.1.251). >2014-04-29 08:57:56,173 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-45) START, SPMGetAllTasksInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 7f64392b >2014-04-29 08:57:56,391 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-45) -- executeIrsBrokerCommand: Attempting on storage pool f03e56af-56f8-4150-9517-271bec2305d9 >2014-04-29 08:57:56,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (pool-4-thread-45) START, HSMGetAllTasksInfoVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a), log id: 2c83976d >2014-04-29 08:57:56,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMGetAllTasksInfoVDSCommand] (pool-4-thread-45) FINISH, HSMGetAllTasksInfoVDSCommand, return: [], log id: 2c83976d >2014-04-29 08:57:56,608 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMGetAllTasksInfoVDSCommand] (pool-4-thread-45) FINISH, SPMGetAllTasksInfoVDSCommand, return: [], log id: 7f64392b >2014-04-29 08:57:56,608 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-45) Discovered no tasks on Storage Pool NFS >2014-04-29 08:58:06,052 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-9) [6cdc0704] Lock Acquired to object EngineLock [exclusiveLocks= key: c0532ddb-5faf-4ab6-b93f-c490c588cae4 value: STORAGE >, sharedLocks= ] >2014-04-29 08:58:06,086 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-50) [6cdc0704] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: c0532ddb-5faf-4ab6-b93f-c490c588cae4 Type: Storage >2014-04-29 08:58:06,091 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-50) [6cdc0704] START, DeactivateStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 703f7fff >2014-04-29 08:58:06,538 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-50) [6cdc0704] FINISH, DeactivateStorageDomainVDSCommand, log id: 703f7fff >2014-04-29 08:58:06,538 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-50) [6cdc0704] Lock freed to object EngineLock [exclusiveLocks= key: c0532ddb-5faf-4ab6-b93f-c490c588cae4 value: STORAGE >, sharedLocks= ] >2014-04-29 08:58:06,541 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-4-thread-46) [6cdc0704] After deactivate treatment vds: 10.34.1.251,pool NFS >2014-04-29 08:58:06,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-46) [6cdc0704] START, RefreshStoragePoolVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 60790bf1 >2014-04-29 08:58:06,544 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-4-thread-47) [6cdc0704] After deactivate treatment vds: 10.34.27.209,pool NFS >2014-04-29 08:58:06,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [6cdc0704] START, RefreshStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 71009e4a >2014-04-29 08:58:06,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-46) [6cdc0704] FINISH, RefreshStoragePoolVDSCommand, log id: 60790bf1 >2014-04-29 08:58:07,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [6cdc0704] Failed in RefreshStoragePoolVDS method >2014-04-29 08:58:07,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [6cdc0704] Error code StoragePoolActionError and error message VDSGenericException: VDSErrorException: Failed to RefreshStoragePoolVDS, error = Error storage pool action: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1',) >2014-04-29 08:58:07,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [6cdc0704] Command org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=300, mMessage=Error storage pool action: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1',)]] >2014-04-29 08:58:07,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [6cdc0704] HostName = 10.34.27.209 >2014-04-29 08:58:07,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [6cdc0704] Command RefreshStoragePoolVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to RefreshStoragePoolVDS, error = Error storage pool action: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1',) >2014-04-29 08:58:07,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-47) [6cdc0704] FINISH, RefreshStoragePoolVDSCommand, log id: 71009e4a >2014-04-29 08:58:07,094 ERROR [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-4-thread-47) [6cdc0704] Failed to refresh storagePool. Host 10.34.27.209 to storage pool NFS. Exception: {3}: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to RefreshStoragePoolVDS, error = Error storage pool action: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1',) (Failed with error StoragePoolActionError and code 300) > at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:122) [bll.jar:] > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] > at org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation.execute(AfterDeactivateSingleAsyncOperation.java:32) [bll.jar:] > at org.ovirt.engine.core.utils.SyncronizeNumberOfAsyncOperations$AsyncOpThread.call(SyncronizeNumberOfAsyncOperations.java:42) [utils.jar:] > at org.ovirt.engine.core.utils.SyncronizeNumberOfAsyncOperations$AsyncOpThread.call(SyncronizeNumberOfAsyncOperations.java:31) [utils.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:99) [utils.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-04-29 08:58:07,096 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-50) [6b1b0287] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 08:58:07,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-50) [6b1b0287] START, DisconnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: df099e2 >2014-04-29 08:58:15,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-50) [6b1b0287] FINISH, DisconnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: df099e2 >2014-04-29 08:58:15,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) [6b1b0287] Correlation ID: 6cdc0704, Job ID: efc9b2ca-af1b-4e76-b8f0-c0e5beaa1a39, Call Stack: null, Custom Event ID: -1, Message: Storage Domain gal (Data Center NFS) was deactivated by admin@internal >2014-04-29 08:58:16,959 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-48) domain 6ba7421b-4b80-4aff-ba01-1c08d3ef4619:iso in problem. vds: 10.34.27.209 >2014-04-29 08:58:16,960 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-48) domain b115b57b-7418-4b60-86b2-64af22fcd1db:storage in problem. vds: 10.34.27.209 >2014-04-29 08:58:19,227 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (ajp-/127.0.0.1:8702-3) [663ed8b6] Lock Acquired to object EngineLock [exclusiveLocks= key: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 value: VM >, sharedLocks= ] >2014-04-29 08:58:19,293 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-50) [663ed8b6] Running command: RemoveVmCommand internal: false. Entities affected : ID: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 Type: VM >2014-04-29 08:58:19,293 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-50) [663ed8b6] START, SetVmStatusVDSCommand( vmId = b066ff4b-970f-4305-94a4-1f32f7fa6fe2, status = ImageLocked), log id: 7576964f >2014-04-29 08:58:19,296 INFO [org.ovirt.engine.core.vdsbroker.SetVmStatusVDSCommand] (pool-4-thread-50) [663ed8b6] FINISH, SetVmStatusVDSCommand, log id: 7576964f >2014-04-29 08:58:19,297 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-50) [663ed8b6] Lock freed to object EngineLock [exclusiveLocks= key: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 value: VM >, sharedLocks= ] >2014-04-29 08:58:19,337 INFO [org.ovirt.engine.core.bll.RemoveAllVmImagesCommand] (pool-4-thread-50) [2d100ff7] Running command: RemoveAllVmImagesCommand internal: true. Entities affected : ID: b066ff4b-970f-4305-94a4-1f32f7fa6fe2 Type: VM >2014-04-29 08:58:19,342 INFO [org.ovirt.engine.core.bll.RemoveImageCommand] (pool-4-thread-50) [3915da01] Running command: RemoveImageCommand internal: true. Entities affected : ID: 00000000-0000-0000-0000-000000000000 Type: Storage >2014-04-29 08:58:19,351 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-50) [3915da01] START, DeleteImageGroupVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, imageGroupId = 431c414f-83e8-4ad7-a08b-050ee71f9784, postZeros = false, forceDelete = false), log id: 5ab17545 >2014-04-29 08:58:19,944 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeleteImageGroupVDSCommand] (pool-4-thread-50) [3915da01] FINISH, DeleteImageGroupVDSCommand, log id: 5ab17545 >2014-04-29 08:58:19,951 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) [3915da01] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command c74a6883-c025-46ce-9bd1-162f45d11461 >2014-04-29 08:58:19,951 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-4-thread-50) [3915da01] CommandMultiAsyncTasks::AttachTask: Attaching task c09756bf-75b8-46d9-8f68-bda208615a68 to command c74a6883-c025-46ce-9bd1-162f45d11461. >2014-04-29 08:58:19,968 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-50) [3915da01] Adding task c09756bf-75b8-46d9-8f68-bda208615a68 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-04-29 08:58:20,037 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) [3915da01] Correlation ID: 663ed8b6, Job ID: 81452ac7-9f9e-4f61-8012-92d84fee5065, Call Stack: null, Custom Event ID: -1, Message: VM Gal was successfully removed. >2014-04-29 08:58:20,037 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) [3915da01] BaseAsyncTask::StartPollingTask: Starting to poll task c09756bf-75b8-46d9-8f68-bda208615a68. >2014-04-29 08:58:28,633 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-9) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-04-29 08:58:28,851 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-9) SPMAsyncTask::PollTask: Polling task c09756bf-75b8-46d9-8f68-bda208615a68 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-04-29 08:58:28,855 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-9) BaseAsyncTask::OnTaskEndSuccess: Task c09756bf-75b8-46d9-8f68-bda208615a68 (Parent Command RemoveVm, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-04-29 08:58:28,855 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-9) CommandAsyncTask::EndActionIfNecessary: All tasks of command c74a6883-c025-46ce-9bd1-162f45d11461 has ended -> executing EndAction >2014-04-29 08:58:28,855 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-9) CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: c74a6883-c025-46ce-9bd1-162f45d11461): calling EndAction . >2014-04-29 08:58:28,855 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-48) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to EndAction RemoveVm, executionIndex: 0 >2014-04-29 08:58:28,863 INFO [org.ovirt.engine.core.bll.RemoveVmCommand] (pool-4-thread-48) Ending command successfully: org.ovirt.engine.core.bll.RemoveVmCommand >2014-04-29 08:58:28,866 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-48) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVm completed, handling the result. >2014-04-29 08:58:28,866 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-48) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type RemoveVm succeeded, clearing tasks. >2014-04-29 08:58:28,868 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-48) SPMAsyncTask::ClearAsyncTask: Attempting to clear task c09756bf-75b8-46d9-8f68-bda208615a68 >2014-04-29 08:58:28,868 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-48) START, SPMClearTaskVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, taskId = c09756bf-75b8-46d9-8f68-bda208615a68), log id: 7b08f856 >2014-04-29 08:58:28,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-48) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=c09756bf-75b8-46d9-8f68-bda208615a68), log id: 9e81535 >2014-04-29 08:58:29,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-48) FINISH, HSMClearTaskVDSCommand, log id: 9e81535 >2014-04-29 08:58:29,108 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-48) FINISH, SPMClearTaskVDSCommand, log id: 7b08f856 >2014-04-29 08:58:29,111 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-48) BaseAsyncTask::RemoveTaskFromDB: Removed task c09756bf-75b8-46d9-8f68-bda208615a68 from DataBase >2014-04-29 08:58:29,111 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-48) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity c74a6883-c025-46ce-9bd1-162f45d11461 >2014-04-29 08:58:38,061 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (ajp-/127.0.0.1:8702-8) MultipleActionsRunner of type RemoveStorageDomain invoked with no actions >2014-04-29 08:58:38,111 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-48) [6f1d2738] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected : ID: c0532ddb-5faf-4ab6-b93f-c490c588cae4 Type: Storage >2014-04-29 08:58:38,111 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-48) [6f1d2738] Start detach storage domain >2014-04-29 08:58:38,112 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-48) [6f1d2738] Detach storage domain: before connect >2014-04-29 08:58:38,119 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-47) [3d4d8fe6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 08:58:38,122 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-50) [4dae77c2] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 08:58:38,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [3d4d8fe6] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 56ef80d2 >2014-04-29 08:58:38,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [4dae77c2] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6a90c422 >2014-04-29 08:58:38,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-47) [3d4d8fe6] FINISH, ConnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 56ef80d2 >2014-04-29 08:58:38,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [4dae77c2] FINISH, ConnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 6a90c422 >2014-04-29 08:58:38,436 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-48) [6f1d2738] Detach storage domain: after connect >2014-04-29 08:58:38,436 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-4-thread-48) [6f1d2738] START, DetachStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = c0532ddb-5faf-4ab6-b93f-c490c588cae4, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 37207504 >2014-04-29 08:58:46,951 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-4-thread-48) [6f1d2738] FINISH, DetachStorageDomainVDSCommand, log id: 37207504 >2014-04-29 08:58:46,951 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-48) [6f1d2738] Detach storage domain: after detach in vds >2014-04-29 08:58:46,958 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [6f1d2738] START, RefreshStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 2965156 >2014-04-29 08:58:46,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-50) [6f1d2738] START, RefreshStoragePoolVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 2f50a422 >2014-04-29 08:58:47,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-50) [6f1d2738] FINISH, RefreshStoragePoolVDSCommand, log id: 2f50a422 >2014-04-29 08:58:47,181 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [6f1d2738] Failed in RefreshStoragePoolVDS method >2014-04-29 08:58:47,181 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [6f1d2738] Error code StoragePoolUnknown and error message VDSGenericException: VDSErrorException: Failed to RefreshStoragePoolVDS, error = Unknown pool id, pool not connected: ('f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 08:58:47,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [6f1d2738] Command org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand return value > StatusOnlyReturnForXmlRpc [mStatus=StatusForXmlRpc [mCode=309, mMessage=Unknown pool id, pool not connected: ('f03e56af-56f8-4150-9517-271bec2305d9',)]] >2014-04-29 08:58:47,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [6f1d2738] HostName = 10.34.27.209 >2014-04-29 08:58:47,181 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [6f1d2738] Command RefreshStoragePoolVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to RefreshStoragePoolVDS, error = Unknown pool id, pool not connected: ('f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 08:58:47,181 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-49) [6f1d2738] FINISH, RefreshStoragePoolVDSCommand, log id: 2965156 >2014-04-29 08:58:47,181 ERROR [org.ovirt.engine.core.bll.storage.RefreshStoragePoolAndDisconnectAsyncOperation] (pool-4-thread-49) [6f1d2738] Failed to connect/refresh storagePool. Host 10.34.27.209 to storage pool NFS. Exception: {3}: org.ovirt.engine.core.common.errors.VdcBLLException: VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to RefreshStoragePoolVDS, error = Unknown pool id, pool not connected: ('f03e56af-56f8-4150-9517-271bec2305d9',) (Failed with error StoragePoolUnknown and code 309) > at org.ovirt.engine.core.bll.VdsHandler.handleVdsResult(VdsHandler.java:122) [bll.jar:] > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) [bll.jar:] > at org.ovirt.engine.core.bll.storage.RefreshStoragePoolAndDisconnectAsyncOperation.execute(RefreshStoragePoolAndDisconnectAsyncOperation.java:27) [bll.jar:] > at org.ovirt.engine.core.utils.SyncronizeNumberOfAsyncOperations$AsyncOpThread.call(SyncronizeNumberOfAsyncOperations.java:42) [utils.jar:] > at org.ovirt.engine.core.utils.SyncronizeNumberOfAsyncOperations$AsyncOpThread.call(SyncronizeNumberOfAsyncOperations.java:31) [utils.jar:] > at org.ovirt.engine.core.utils.threadpool.ThreadPoolUtil$InternalCallable.call(ThreadPoolUtil.java:99) [utils.jar:] > at java.util.concurrent.FutureTask.run(FutureTask.java:262) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_51] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_51] > at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_51] > >2014-04-29 08:58:47,198 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-50) [29f92e55] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 08:58:47,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-50) [29f92e55] START, DisconnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: bf77bfac-76ab-4fdb-99ee-db5e6e34b22d, connection: 10.34.27.209:/mnt/gal, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 8a745bd >2014-04-29 08:58:55,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-50) [29f92e55] FINISH, DisconnectStorageServerVDSCommand, return: {bf77bfac-76ab-4fdb-99ee-db5e6e34b22d=0}, log id: 8a745bd >2014-04-29 08:58:55,640 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-48) [6f1d2738] Detach storage domain: after disconnect storage >2014-04-29 08:58:55,643 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-48) [6f1d2738] End detach storage domain >2014-04-29 08:58:55,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-48) [6f1d2738] Correlation ID: 6f1d2738, Job ID: d888b5c0-ac0e-469b-862b-3656365a9040, Call Stack: null, Custom Event ID: -1, Message: Storage Domain gal was detached from Data Center NFS by admin@internal >2014-04-29 08:59:25,768 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-3) [2aa418e0] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.1.251:/mnt/export2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-04-29 08:59:25,774 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-3) [2aa418e0] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 08:59:25,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-3) [2aa418e0] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3537a949 >2014-04-29 08:59:26,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-3) [2aa418e0] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=477}, log id: 3537a949 >2014-04-29 08:59:26,063 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [2aa418e0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection 10.34.1.251:/mnt/export2 returned by VDSM was: Problem while trying to mount target >2014-04-29 08:59:26,063 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (ajp-/127.0.0.1:8702-3) [2aa418e0] The connection with details 10.34.1.251:/mnt/export2 failed because of error code 477 and error message is: problem while trying to mount target >2014-04-29 08:59:26,067 ERROR [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-3) [2aa418e0] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand. >2014-04-29 08:59:26,076 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-3) [2aa418e0] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.1.251:/mnt/export2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-04-29 08:59:58,341 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [680fa10f] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.1.251:/mnt/export2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-04-29 08:59:58,353 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [680fa10f] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 08:59:58,354 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [680fa10f] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 403f8d6f >2014-04-29 08:59:58,634 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-59) Setting new tasks map. The map contains now 0 tasks >2014-04-29 08:59:58,634 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-59) Cleared all tasks of pool f03e56af-56f8-4150-9517-271bec2305d9. >2014-04-29 08:59:58,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [680fa10f] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 403f8d6f >2014-04-29 08:59:58,693 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [680fa10f] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.1.251:/mnt/export2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-04-29 08:59:58,722 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [353d1704] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 08:59:58,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [353d1704] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3320e20e >2014-04-29 08:59:58,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-2) [353d1704] FINISH, ConnectStorageServerVDSCommand, return: {8925e659-8b90-40f7-8e79-184a928c496f=0}, log id: 3320e20e >2014-04-29 08:59:58,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-2) [353d1704] START, CreateStorageDomainVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@fcbed90c, args=10.34.1.251:/mnt/export2), log id: fbb5b8 >2014-04-29 09:00:07,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-2) [353d1704] FINISH, CreateStorageDomainVDSCommand, log id: fbb5b8 >2014-04-29 09:00:07,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-2) [353d1704] START, GetStorageDomainStatsVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storageDomainId=e4395463-7e5d-4c77-9dba-8e2edf383ef8), log id: 63a7e006 >2014-04-29 09:00:07,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-2) [353d1704] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@21b25298, log id: 63a7e006 >2014-04-29 09:00:07,381 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-2) [353d1704] Correlation ID: 353d1704, Job ID: 9c724dca-297b-4b8f-a12f-8157096936d9, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export2 was added by admin@internal >2014-04-29 09:00:07,472 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-1) [7de3944a] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:00:07,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [7de3944a] START, DisconnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: f58666c >2014-04-29 09:00:12,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [7de3944a] FINISH, DisconnectStorageServerVDSCommand, return: {8925e659-8b90-40f7-8e79-184a928c496f=0}, log id: f58666c >2014-04-29 09:00:12,993 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-3) [63db7864] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: e4395463-7e5d-4c77-9dba-8e2edf383ef8 Type: Storage >2014-04-29 09:00:13,001 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-48) [5dca0a94] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:00:13,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [5dca0a94] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1674ee95 >2014-04-29 09:00:13,005 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-46) [44579332] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:00:13,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-46) [44579332] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 67ac8f17 >2014-04-29 09:00:13,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-48) [5dca0a94] FINISH, ConnectStorageServerVDSCommand, return: {8925e659-8b90-40f7-8e79-184a928c496f=0}, log id: 1674ee95 >2014-04-29 09:02:18,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [44579332] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to the Storage Domains export2. >2014-04-29 09:02:18,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-46) [44579332] FINISH, ConnectStorageServerVDSCommand, return: {8925e659-8b90-40f7-8e79-184a928c496f=477}, log id: 67ac8f17 >2014-04-29 09:02:18,405 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [44579332] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection 10.34.1.251:/mnt/export2 returned by VDSM was: Problem while trying to mount target >2014-04-29 09:02:18,405 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (pool-4-thread-46) [44579332] The connection with details 10.34.1.251:/mnt/export2 failed because of error code 477 and error message is: problem while trying to mount target >2014-04-29 09:02:18,408 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-46) [44579332] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-04-29 09:02:18,408 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [63db7864] START, AttachStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = e4395463-7e5d-4c77-9dba-8e2edf383ef8), log id: 444cf9ed >2014-04-29 09:02:47,482 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [63db7864] FINISH, AttachStorageDomainVDSCommand, log id: 444cf9ed >2014-04-29 09:02:47,486 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [49b09472] Lock Acquired to object EngineLock [exclusiveLocks= key: e4395463-7e5d-4c77-9dba-8e2edf383ef8 value: STORAGE >, sharedLocks= ] >2014-04-29 09:02:47,498 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [49b09472] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: e4395463-7e5d-4c77-9dba-8e2edf383ef8 Type: Storage >2014-04-29 09:02:47,501 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [49b09472] Lock freed to object EngineLock [exclusiveLocks= key: e4395463-7e5d-4c77-9dba-8e2edf383ef8 value: STORAGE >, sharedLocks= ] >2014-04-29 09:02:47,501 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [49b09472] ActivateStorage Domain. Before Connect all hosts to pool. Time:4/29/14 9:02 AM >2014-04-29 09:02:47,507 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-46) [7b8581c0] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:02:47,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-46) [7b8581c0] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 411b0aa0 >2014-04-29 09:02:47,511 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-50) [6ca742d6] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:02:47,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [6ca742d6] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1a8ff9fb >2014-04-29 09:02:47,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [6ca742d6] FINISH, ConnectStorageServerVDSCommand, return: {8925e659-8b90-40f7-8e79-184a928c496f=0}, log id: 1a8ff9fb >2014-04-29 09:03:16,962 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-49) starting ProcessDomainRecovery for domain b115b57b-7418-4b60-86b2-64af22fcd1db:storage >2014-04-29 09:03:16,999 WARN [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-49) vds 10.34.27.209 reported domain b115b57b-7418-4b60-86b2-64af22fcd1db:storage as in problem, attempting to move the vds to status NonOperational >2014-04-29 09:03:17,002 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.IrsBrokerCommand] (pool-4-thread-49) Domain 6ba7421b-4b80-4aff-ba01-1c08d3ef4619:iso has recovered from problem. No active host in the DC is reporting it as poblematic, so clearing the domain recovery timer. >2014-04-29 09:03:17,015 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (pool-4-thread-48) [3b8b773e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:03:17,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [3b8b773e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3df107a5 >2014-04-29 09:03:17,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (pool-4-thread-48) [3b8b773e] FINISH, SetVdsStatusVDSCommand, log id: 3df107a5 >2014-04-29 09:03:17,025 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-48) [3b8b773e] Correlation ID: 3b8b773e, Job ID: a3f40ffd-5ab2-40ad-8e33-3543e9b7f6b5, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:04:52,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [7b8581c0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to the Storage Domains export2. >2014-04-29 09:04:52,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-46) [7b8581c0] FINISH, ConnectStorageServerVDSCommand, return: {8925e659-8b90-40f7-8e79-184a928c496f=477}, log id: 411b0aa0 >2014-04-29 09:04:52,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [7b8581c0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection 10.34.1.251:/mnt/export2 returned by VDSM was: Problem while trying to mount target >2014-04-29 09:04:52,793 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (pool-4-thread-46) [7b8581c0] The connection with details 10.34.1.251:/mnt/export2 failed because of error code 477 and error message is: problem while trying to mount target >2014-04-29 09:04:52,795 ERROR [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-46) [7b8581c0] Transaction rolled-back for command: org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand. >2014-04-29 09:04:52,795 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [49b09472] START, ActivateStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = e4395463-7e5d-4c77-9dba-8e2edf383ef8), log id: 7514f238 >2014-04-29 09:04:53,176 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [49b09472] FINISH, ActivateStorageDomainVDSCommand, log id: 7514f238 >2014-04-29 09:04:53,176 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [49b09472] ActivateStorage Domain. After Connect all hosts to pool. Time:4/29/14 9:04 AM >2014-04-29 09:04:53,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-45) [49b09472] START, RefreshStoragePoolVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: de9cd9 >2014-04-29 09:04:53,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-45) [49b09472] FINISH, RefreshStoragePoolVDSCommand, log id: de9cd9 >2014-04-29 09:04:53,406 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (pool-4-thread-45) [49b09472] Refreshed vds 10.34.1.251 in pool NFS >2014-04-29 09:04:53,408 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [49b09472] ActivateStorage Domain. After change storage pool status in vds. Time:4/29/14 9:04 AM >2014-04-29 09:04:53,411 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [49b09472] Correlation ID: 49b09472, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export2 (Data Center NFS) was activated by admin@internal >2014-04-29 09:04:53,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [49b09472] Correlation ID: 63db7864, Job ID: e8057617-2f45-4545-bac4-a7fe963ec0c7, Call Stack: null, Custom Event ID: -1, Message: Storage Domain export2 was attached to Data Center NFS by admin@internal >2014-04-29 09:05:00,014 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-18) Autorecovering 1 hosts >2014-04-29 09:05:00,016 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-18) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:05:00,020 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:05:00,020 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:05:00,020 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:05:00,020 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:05:00,024 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7e42bbae >2014-04-29 09:05:00,027 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] FINISH, SetVdsStatusVDSCommand, log id: 7e42bbae >2014-04-29 09:05:00,028 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 187cff91 >2014-04-29 09:05:00,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4c396af7 >2014-04-29 09:05:00,961 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] Failed in GetHardwareInfoVDS method >2014-04-29 09:05:00,961 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:05:00,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4d9da539 >2014-04-29 09:05:00,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] HostName = 10.34.27.209 >2014-04-29 09:05:00,961 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:05:00,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4c53b651] FINISH, GetHardwareInfoVDSCommand, log id: 4c396af7 >2014-04-29 09:05:00,966 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [4c53b651] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:05:00,977 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-18) [261720] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:05:00,982 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:05:00,983 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 187cff91 >2014-04-29 09:05:01,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:05:01,012 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [6906c39a] Correlation ID: 4c53b651, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was autorecovered. >2014-04-29 09:05:01,012 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:05:01,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 40a3ff97 >2014-04-29 09:05:01,624 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) Failed in GetHardwareInfoVDS method >2014-04-29 09:05:01,624 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:05:01,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4ee133a8 >2014-04-29 09:05:01,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) HostName = 10.34.27.209 >2014-04-29 09:05:01,624 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:05:01,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) FINISH, GetHardwareInfoVDSCommand, log id: 40a3ff97 >2014-04-29 09:05:01,634 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:05:01,649 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:05:01,866 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-34) [5f10e805] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:05:01,867 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [418875d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:05:01,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [418875d8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64d64a5b >2014-04-29 09:07:07,204 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [418875d8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to the Storage Domains export2. >2014-04-29 09:07:07,204 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [418875d8] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0, 854c3686-561c-4079-af7a-2115dab8d0b5=0, 8925e659-8b90-40f7-8e79-184a928c496f=477}, log id: 64d64a5b >2014-04-29 09:07:07,234 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [418875d8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection 10.34.1.251:/mnt/export2 returned by VDSM was: Problem while trying to mount target >2014-04-29 09:07:07,235 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (DefaultQuartzScheduler_Worker-34) [418875d8] The connection with details 10.34.1.251:/mnt/export2 failed because of error code 477 and error message is: problem while trying to mount target >2014-04-29 09:07:07,235 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [418875d8] Host 10.34.27.209 storage connection was failed >2014-04-29 09:07:07,264 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [418875d8] Correlation ID: 418875d8, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Servers >2014-04-29 09:07:07,269 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 14774536 >2014-04-29 09:07:08,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 14774536 >2014-04-29 09:07:08,488 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:07:08,502 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-34) [46bcca9d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:07:08,504 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [46bcca9d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 518add47 >2014-04-29 09:07:08,507 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [46bcca9d] FINISH, SetVdsStatusVDSCommand, log id: 518add47 >2014-04-29 09:07:08,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [46bcca9d] Correlation ID: 46bcca9d, Job ID: b763df2e-71b1-4e49-9e66-04141b421d6e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:07:08,524 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [46bcca9d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:07:08,527 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [46bcca9d] Correlation ID: 5f10e805, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:07:08,531 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [171fb79a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:07:08,533 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [64c6b82] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:07:08,534 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [64c6b82] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 09:10:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-30) [1f881e49] Autorecovering 1 hosts >2014-04-29 09:10:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-30) [1f881e49] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:10:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5f0d7c56 >2014-04-29 09:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] FINISH, SetVdsStatusVDSCommand, log id: 5f0d7c56 >2014-04-29 09:10:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 46f013ef >2014-04-29 09:10:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1503c51d >2014-04-29 09:10:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] Failed in GetHardwareInfoVDS method >2014-04-29 09:10:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:10:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5b127d4e >2014-04-29 09:10:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] HostName = 10.34.27.209 >2014-04-29 09:10:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:10:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [1fff65e9] FINISH, GetHardwareInfoVDSCommand, log id: 1503c51d >2014-04-29 09:10:00,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [1fff65e9] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:10:00,721 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-30) [27727366] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:10:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-30) [1b655e9d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:10:00,723 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) [1b655e9d] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 46f013ef >2014-04-29 09:10:00,741 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [1b655e9d] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:10:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [1b655e9d] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:10:03,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2bade771 >2014-04-29 09:10:03,410 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) Failed in GetHardwareInfoVDS method >2014-04-29 09:10:03,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:10:03,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2635c21e >2014-04-29 09:10:03,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) HostName = 10.34.27.209 >2014-04-29 09:10:03,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:10:03,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) FINISH, GetHardwareInfoVDSCommand, log id: 2bade771 >2014-04-29 09:10:03,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:10:03,431 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:10:03,650 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-25) [f559ae2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:10:03,652 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [3f46b867] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:10:03,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [3f46b867] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6ba3a996 >2014-04-29 09:10:43,836 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-11) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = e4395463-7e5d-4c77-9dba-8e2edf383ef8, vmIdList = null), log id: 38659066 >2014-04-29 09:10:49,077 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-11) FINISH, GetVmsInfoVDSCommand, log id: 38659066 >2014-04-29 09:10:50,396 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-2) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = e4395463-7e5d-4c77-9dba-8e2edf383ef8, vmIdList = null), log id: 3c2e8780 >2014-04-29 09:10:50,618 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-2) FINISH, GetVmsInfoVDSCommand, log id: 3c2e8780 >2014-04-29 09:12:08,954 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [3f46b867] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to the Storage Domains export2. >2014-04-29 09:12:08,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [3f46b867] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0, 854c3686-561c-4079-af7a-2115dab8d0b5=0, 8925e659-8b90-40f7-8e79-184a928c496f=477}, log id: 6ba3a996 >2014-04-29 09:12:08,957 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [3f46b867] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: The error message for connection 10.34.1.251:/mnt/export2 returned by VDSM was: Problem while trying to mount target >2014-04-29 09:12:08,957 ERROR [org.ovirt.engine.core.bll.storage.NFSStorageHelper] (DefaultQuartzScheduler_Worker-25) [3f46b867] The connection with details 10.34.1.251:/mnt/export2 failed because of error code 477 and error message is: problem while trying to mount target >2014-04-29 09:12:08,957 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [3f46b867] Host 10.34.27.209 storage connection was failed >2014-04-29 09:12:08,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [3f46b867] Correlation ID: 3f46b867, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Servers >2014-04-29 09:12:08,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 8b19526 >2014-04-29 09:12:09,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 8b19526 >2014-04-29 09:12:09,396 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:12:09,411 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-25) [53fb03c1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:12:09,412 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [53fb03c1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: f001d5b >2014-04-29 09:12:09,414 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [53fb03c1] FINISH, SetVdsStatusVDSCommand, log id: f001d5b >2014-04-29 09:12:09,420 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [53fb03c1] Correlation ID: 53fb03c1, Job ID: d7f668c6-a0b3-4275-9e73-fc2bc910f5cc, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:12:09,430 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [53fb03c1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:12:09,432 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [53fb03c1] Correlation ID: f559ae2, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:12:09,435 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-25) [4947c73] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:12:09,436 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-25) [439a91a9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:12:09,437 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [439a91a9] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 09:12:24,876 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [5557483c] Lock Acquired to object EngineLock [exclusiveLocks= key: 10.34.1.251:/mnt/storage2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-04-29 09:12:24,884 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [5557483c] Running command: AddStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:12:24,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [5557483c] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: null, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 746fb274 >2014-04-29 09:12:25,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-5) [5557483c] FINISH, ConnectStorageServerVDSCommand, return: {00000000-0000-0000-0000-000000000000=0}, log id: 746fb274 >2014-04-29 09:12:25,335 INFO [org.ovirt.engine.core.bll.storage.AddStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-5) [5557483c] Lock freed to object EngineLock [exclusiveLocks= key: 10.34.1.251:/mnt/storage2 value: STORAGE_CONNECTION >, sharedLocks= ] >2014-04-29 09:12:25,363 INFO [org.ovirt.engine.core.bll.storage.AddNFSStorageDomainCommand] (ajp-/127.0.0.1:8702-1) [8b8a77c] Running command: AddNFSStorageDomainCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:12:25,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [8b8a77c] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 457a1d7d >2014-04-29 09:12:25,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-1) [8b8a77c] FINISH, ConnectStorageServerVDSCommand, return: {19363eb9-2267-4f2b-9c03-2035530f41d2=0}, log id: 457a1d7d >2014-04-29 09:12:25,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [8b8a77c] START, CreateStorageDomainVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storageDomain=org.ovirt.engine.core.common.businessentities.StorageDomainStatic@9eb900f0, args=10.34.1.251:/mnt/storage2), log id: e3560cf >2014-04-29 09:12:33,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.CreateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-1) [8b8a77c] FINISH, CreateStorageDomainVDSCommand, log id: e3560cf >2014-04-29 09:12:33,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-1) [8b8a77c] START, GetStorageDomainStatsVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storageDomainId=522919d2-161c-403b-ba40-b670fa42994b), log id: 264bc06 >2014-04-29 09:12:33,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetStorageDomainStatsVDSCommand] (ajp-/127.0.0.1:8702-1) [8b8a77c] FINISH, GetStorageDomainStatsVDSCommand, return: org.ovirt.engine.core.common.businessentities.StorageDomain@32b41ea6, log id: 264bc06 >2014-04-29 09:12:33,418 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-1) [8b8a77c] Correlation ID: 8b8a77c, Job ID: 840a54f3-fbb3-47ed-8ac2-815488f7bdf0, Call Stack: null, Custom Event ID: -1, Message: Storage Domain storage2 was added by admin@internal >2014-04-29 09:12:33,451 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (ajp-/127.0.0.1:8702-10) [7ec7f798] Running command: DisconnectStorageServerConnectionCommand internal: false. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:12:33,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [7ec7f798] START, DisconnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2eabf492 >2014-04-29 09:12:39,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (ajp-/127.0.0.1:8702-10) [7ec7f798] FINISH, DisconnectStorageServerVDSCommand, return: {19363eb9-2267-4f2b-9c03-2035530f41d2=0}, log id: 2eabf492 >2014-04-29 09:12:39,991 INFO [org.ovirt.engine.core.bll.storage.AttachStorageDomainToPoolCommand] (ajp-/127.0.0.1:8702-3) [5190d63b] Running command: AttachStorageDomainToPoolCommand internal: false. Entities affected : ID: 522919d2-161c-403b-ba40-b670fa42994b Type: Storage >2014-04-29 09:12:39,998 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-46) [2ea922e3] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:12:40,000 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-46) [2ea922e3] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2c22f673 >2014-04-29 09:12:40,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-46) [2ea922e3] FINISH, ConnectStorageServerVDSCommand, return: {19363eb9-2267-4f2b-9c03-2035530f41d2=0}, log id: 2c22f673 >2014-04-29 09:12:40,302 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [5190d63b] START, AttachStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = 522919d2-161c-403b-ba40-b670fa42994b), log id: 3acbbe90 >2014-04-29 09:13:10,994 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.AttachStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [5190d63b] FINISH, AttachStorageDomainVDSCommand, log id: 3acbbe90 >2014-04-29 09:13:10,997 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [125c1dfa] Lock Acquired to object EngineLock [exclusiveLocks= key: 522919d2-161c-403b-ba40-b670fa42994b value: STORAGE >, sharedLocks= ] >2014-04-29 09:13:11,005 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [125c1dfa] Running command: ActivateStorageDomainCommand internal: true. Entities affected : ID: 522919d2-161c-403b-ba40-b670fa42994b Type: Storage >2014-04-29 09:13:11,008 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [125c1dfa] Lock freed to object EngineLock [exclusiveLocks= key: 522919d2-161c-403b-ba40-b670fa42994b value: STORAGE >, sharedLocks= ] >2014-04-29 09:13:11,008 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [125c1dfa] ActivateStorage Domain. Before Connect all hosts to pool. Time:4/29/14 9:13 AM >2014-04-29 09:13:11,011 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-45) [3d0861b5] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:13:11,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-45) [3d0861b5] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64432987 >2014-04-29 09:13:11,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-45) [3d0861b5] FINISH, ConnectStorageServerVDSCommand, return: {19363eb9-2267-4f2b-9c03-2035530f41d2=0}, log id: 64432987 >2014-04-29 09:13:11,235 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [125c1dfa] START, ActivateStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = 522919d2-161c-403b-ba40-b670fa42994b), log id: 7796391f >2014-04-29 09:13:18,986 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.ActivateStorageDomainVDSCommand] (ajp-/127.0.0.1:8702-3) [125c1dfa] FINISH, ActivateStorageDomainVDSCommand, log id: 7796391f >2014-04-29 09:13:18,986 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [125c1dfa] ActivateStorage Domain. After Connect all hosts to pool. Time:4/29/14 9:13 AM >2014-04-29 09:13:18,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-46) [125c1dfa] START, RefreshStoragePoolVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 74cdb7f9 >2014-04-29 09:13:19,213 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-46) [125c1dfa] FINISH, RefreshStoragePoolVDSCommand, log id: 74cdb7f9 >2014-04-29 09:13:19,213 INFO [org.ovirt.engine.core.bll.storage.RefreshPoolSingleAsyncOperation] (pool-4-thread-46) [125c1dfa] Refreshed vds 10.34.1.251 in pool NFS >2014-04-29 09:13:19,215 INFO [org.ovirt.engine.core.bll.storage.ActivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [125c1dfa] ActivateStorage Domain. After change storage pool status in vds. Time:4/29/14 9:13 AM >2014-04-29 09:13:19,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [125c1dfa] Correlation ID: 125c1dfa, Call Stack: null, Custom Event ID: -1, Message: Storage Domain storage2 (Data Center NFS) was activated by admin@internal >2014-04-29 09:13:19,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (ajp-/127.0.0.1:8702-3) [125c1dfa] Correlation ID: 5190d63b, Job ID: 813bc5c5-93ff-4542-a379-638ba8ea5497, Call Stack: null, Custom Event ID: -1, Message: Storage Domain storage2 was attached to Data Center NFS by admin@internal >2014-04-29 09:14:00,059 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-4) [5199bc16] START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = e4395463-7e5d-4c77-9dba-8e2edf383ef8, vmIdList = null), log id: 67fb7251 >2014-04-29 09:14:00,285 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-4) [5199bc16] FINISH, GetVmsInfoVDSCommand, log id: 67fb7251 >2014-04-29 09:14:00,498 INFO [org.ovirt.engine.core.bll.ImportVmTemplateCommand] (pool-4-thread-46) [5199bc16] Running command: ImportVmTemplateCommand internal: false. Entities affected : ID: 522919d2-161c-403b-ba40-b670fa42994b Type: Storage >2014-04-29 09:14:00,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [5199bc16] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: While importing Template sswe_prd_v03.0.0, the Network/s VM_Data_4 were found to be Non-VM Networks or do not exist in Cluster. Network Name was not set in the Interface/s nic1. >2014-04-29 09:14:00,526 INFO [org.ovirt.engine.core.bll.CopyImageGroupCommand] (pool-4-thread-46) [17c0ce43] Running command: CopyImageGroupCommand internal: true. Entities affected : ID: 522919d2-161c-403b-ba40-b670fa42994b Type: Storage >2014-04-29 09:14:00,555 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-10) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = e4395463-7e5d-4c77-9dba-8e2edf383ef8, vmIdList = null), log id: 45ed2b17 >2014-04-29 09:14:00,574 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-46) [17c0ce43] START, CopyImageVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = e4395463-7e5d-4c77-9dba-8e2edf383ef8, imageGroupId = 43f667af-cd78-4e40-b5a9-004bd91bb406, imageId = dfde1f5e-cd6f-44ba-8af5-84d52055a756, dstImageGroupId = 43f667af-cd78-4e40-b5a9-004bd91bb406, vmId = 1f5a294b-d418-49b4-a5d6-0f6f90eebfb1, dstImageId = dfde1f5e-cd6f-44ba-8af5-84d52055a756, imageDescription = , dstStorageDomainId = 522919d2-161c-403b-ba40-b670fa42994b, copyVolumeType = SharedVol, volumeFormat = RAW, preallocate = Sparse, postZero = false, force = false), log id: 4a1a6a33 >2014-04-29 09:14:00,776 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-10) FINISH, GetVmsInfoVDSCommand, log id: 45ed2b17 >2014-04-29 09:14:00,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-46) [17c0ce43] -- executeIrsBrokerCommand: calling 'copyImage' with two new parameters: description and UUID >2014-04-29 09:14:00,779 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-46) [17c0ce43] -- copyImage parameters: > sdUUID=e4395463-7e5d-4c77-9dba-8e2edf383ef8 > spUUID=f03e56af-56f8-4150-9517-271bec2305d9 > vmGUID=1f5a294b-d418-49b4-a5d6-0f6f90eebfb1 > srcImageGUID=43f667af-cd78-4e40-b5a9-004bd91bb406 > srcVolUUID=dfde1f5e-cd6f-44ba-8af5-84d52055a756 > dstImageGUID=43f667af-cd78-4e40-b5a9-004bd91bb406 > dstVolUUID=dfde1f5e-cd6f-44ba-8af5-84d52055a756 > descr= > dstSdUUID=522919d2-161c-403b-ba40-b670fa42994b >2014-04-29 09:14:01,309 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.CopyImageVDSCommand] (pool-4-thread-46) [17c0ce43] FINISH, CopyImageVDSCommand, return: 00000000-0000-0000-0000-000000000000, log id: 4a1a6a33 >2014-04-29 09:14:01,379 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-46) [17c0ce43] CommandAsyncTask::Adding CommandMultiAsyncTasks object for command af550ca5-d89b-477e-9825-9eeaaa24bf2c >2014-04-29 09:14:01,379 INFO [org.ovirt.engine.core.bll.CommandMultiAsyncTasks] (pool-4-thread-46) [17c0ce43] CommandMultiAsyncTasks::AttachTask: Attaching task c5b5d4d0-5d7b-438a-849b-df35e9ed50ac to command af550ca5-d89b-477e-9825-9eeaaa24bf2c. >2014-04-29 09:14:01,407 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (pool-4-thread-46) [17c0ce43] Adding task c5b5d4d0-5d7b-438a-849b-df35e9ed50ac (Parent Command ImportVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters), polling hasn't started yet.. >2014-04-29 09:14:01,445 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [17c0ce43] Correlation ID: 5199bc16, Job ID: cc2fa90c-b93e-4c16-a590-e821b06c0ca0, Call Stack: null, Custom Event ID: -1, Message: Starting to import Template sswe_prd_v03.0.0 to Data Center NFS, Cluster NFS >2014-04-29 09:14:01,445 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-46) [17c0ce43] BaseAsyncTask::StartPollingTask: Starting to poll task c5b5d4d0-5d7b-438a-849b-df35e9ed50ac. >2014-04-29 09:14:08,884 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-49) Polling and updating Async Tasks: 1 tasks, 1 tasks to poll now >2014-04-29 09:14:09,119 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) SPMAsyncTask::PollTask: Polling task c5b5d4d0-5d7b-438a-849b-df35e9ed50ac (Parent Command ImportVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status running. >2014-04-29 09:14:09,119 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-49) Finished polling Tasks, will poll again in 10 seconds. >2014-04-29 09:15:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-2) [ff45100] Autorecovering 1 hosts >2014-04-29 09:15:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-2) [ff45100] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3b9c2480 >2014-04-29 09:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] FINISH, SetVdsStatusVDSCommand, log id: 3b9c2480 >2014-04-29 09:15:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7840867e >2014-04-29 09:15:00,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5d10c545 >2014-04-29 09:15:00,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] Failed in GetHardwareInfoVDS method >2014-04-29 09:15:00,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:15:00,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@504c8bfa >2014-04-29 09:15:00,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] HostName = 10.34.27.209 >2014-04-29 09:15:00,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:15:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [796ce1da] FINISH, GetHardwareInfoVDSCommand, log id: 5d10c545 >2014-04-29 09:15:00,700 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [796ce1da] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:15:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-2) [41a60995] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:15:00,713 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:15:00,713 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7840867e >2014-04-29 09:15:00,738 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:15:00,738 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:15:03,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1c7635c >2014-04-29 09:15:04,061 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) Failed in GetHardwareInfoVDS method >2014-04-29 09:15:04,062 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:15:04,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@494c2ded >2014-04-29 09:15:04,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) HostName = 10.34.27.209 >2014-04-29 09:15:04,062 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:15:04,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) FINISH, GetHardwareInfoVDSCommand, log id: 1c7635c >2014-04-29 09:15:04,067 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:15:04,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:15:04,295 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-72) [2abab8ed] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:15:04,297 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [4cc3199d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:15:04,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [4cc3199d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70094ce1 >2014-04-29 09:18:04,314 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [4cc3199d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 09:18:04,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [4cc3199d] FINISH, ConnectStorageServerVDSCommand, log id: 70094ce1 >2014-04-29 09:18:04,314 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [4cc3199d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 09:18:04,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 45ecda98 >2014-04-29 09:18:04,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 45ecda98 >2014-04-29 09:18:04,749 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:18:04,777 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-72) [2ee2a63c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:18:04,778 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [2ee2a63c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 684b8 >2014-04-29 09:18:04,781 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [2ee2a63c] FINISH, SetVdsStatusVDSCommand, log id: 684b8 >2014-04-29 09:18:04,790 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [2ee2a63c] Correlation ID: 2ee2a63c, Job ID: 53dfe612-b543-424d-929c-3905b35f8882, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:18:04,797 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [2ee2a63c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:18:04,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [2ee2a63c] Correlation ID: 2abab8ed, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:18:04,804 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-72) [4efdb1e5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:18:04,806 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-72) [27199a6d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:18:04,807 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [27199a6d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 09:18:06,020 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [1a6ca287] Lock Acquired to object EngineLock [exclusiveLocks= key: e4395463-7e5d-4c77-9dba-8e2edf383ef8 value: STORAGE >, sharedLocks= ] >2014-04-29 09:18:06,044 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [1a6ca287] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ERROR_CANNOT_DEACTIVATE_DOMAIN_WITH_TASKS >2014-04-29 09:18:06,044 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-3) [1a6ca287] Lock freed to object EngineLock [exclusiveLocks= key: e4395463-7e5d-4c77-9dba-8e2edf383ef8 value: STORAGE >, sharedLocks= ] >2014-04-29 09:18:21,728 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-2) [6fa1952f] Lock Acquired to object EngineLock [exclusiveLocks= key: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 value: STORAGE >, sharedLocks= ] >2014-04-29 09:18:21,757 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-46) [6fa1952f] Running command: DeactivateStorageDomainCommand internal: false. Entities affected : ID: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 Type: Storage >2014-04-29 09:18:21,763 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-46) [6fa1952f] START, DeactivateStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = 6ba7421b-4b80-4aff-ba01-1c08d3ef4619, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1), log id: 60d21358 >2014-04-29 09:18:22,416 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DeactivateStorageDomainVDSCommand] (pool-4-thread-46) [6fa1952f] FINISH, DeactivateStorageDomainVDSCommand, log id: 60d21358 >2014-04-29 09:18:22,417 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (pool-4-thread-46) [6fa1952f] Lock freed to object EngineLock [exclusiveLocks= key: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 value: STORAGE >, sharedLocks= ] >2014-04-29 09:18:22,419 INFO [org.ovirt.engine.core.bll.storage.AfterDeactivateSingleAsyncOperation] (pool-4-thread-50) [6fa1952f] After deactivate treatment vds: 10.34.1.251,pool NFS >2014-04-29 09:18:22,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-50) [6fa1952f] START, RefreshStoragePoolVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 81e68d2 >2014-04-29 09:18:22,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-50) [6fa1952f] FINISH, RefreshStoragePoolVDSCommand, log id: 81e68d2 >2014-04-29 09:18:22,643 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-46) [3048b459] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:18:22,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-46) [3048b459] START, DisconnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 460dd81d >2014-04-29 09:18:32,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-46) [3048b459] FINISH, DisconnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 460dd81d >2014-04-29 09:18:32,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [3048b459] Correlation ID: 6fa1952f, Job ID: b040f4d5-377a-41a0-aeb1-2c371c4a18ab, Call Stack: null, Custom Event ID: -1, Message: Storage Domain iso (Data Center NFS) was deactivated by admin@internal >2014-04-29 09:18:35,261 INFO [org.ovirt.engine.core.bll.MultipleActionsRunner] (ajp-/127.0.0.1:8702-11) MultipleActionsRunner of type RemoveStorageDomain invoked with no actions >2014-04-29 09:18:35,307 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [24b2854] Running command: DetachStorageDomainFromPoolCommand internal: false. Entities affected : ID: 6ba7421b-4b80-4aff-ba01-1c08d3ef4619 Type: Storage >2014-04-29 09:18:35,307 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [24b2854] Start detach storage domain >2014-04-29 09:18:35,311 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [24b2854] Detach storage domain: before connect >2014-04-29 09:18:35,313 INFO [org.ovirt.engine.core.bll.storage.ConnectStorageToVdsCommand] (pool-4-thread-50) [2a26bdf] Running command: ConnectStorageToVdsCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:18:35,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [2a26bdf] START, ConnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 54ad7f80 >2014-04-29 09:18:35,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (pool-4-thread-50) [2a26bdf] FINISH, ConnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: 54ad7f80 >2014-04-29 09:18:35,695 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [24b2854] Detach storage domain: after connect >2014-04-29 09:18:35,695 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-4-thread-46) [24b2854] START, DetachStorageDomainVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = 6ba7421b-4b80-4aff-ba01-1c08d3ef4619, masterDomainId = 00000000-0000-0000-0000-000000000000, masterVersion = 1, force = false), log id: 70856957 >2014-04-29 09:18:56,010 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-7) START, GetVmsInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = e4395463-7e5d-4c77-9dba-8e2edf383ef8, vmIdList = null), log id: 71bce9be >2014-04-29 09:19:04,751 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.DetachStorageDomainVDSCommand] (pool-4-thread-46) [24b2854] FINISH, DetachStorageDomainVDSCommand, log id: 70856957 >2014-04-29 09:19:04,751 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [24b2854] Detach storage domain: after detach in vds >2014-04-29 09:19:04,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-45) [24b2854] START, RefreshStoragePoolVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, masterStorageDomainId=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1), log id: 3e43bd8a >2014-04-29 09:19:04,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.RefreshStoragePoolVDSCommand] (pool-4-thread-45) [24b2854] FINISH, RefreshStoragePoolVDSCommand, log id: 3e43bd8a >2014-04-29 09:19:04,987 INFO [org.ovirt.engine.core.bll.storage.DisconnectStorageServerConnectionCommand] (pool-4-thread-45) [1019a386] Running command: DisconnectStorageServerConnectionCommand internal: true. Entities affected : ID: aaa00000-0000-0000-0000-123456789aaa Type: System >2014-04-29 09:19:04,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-45) [1019a386] START, DisconnectStorageServerVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, storagePoolId = 00000000-0000-0000-0000-000000000000, storageType = NFS, connectionList = [{ id: 1c52ac83-65c6-49de-9380-1f13ba3e4cc3, connection: 10.34.27.209:/mnt/iso, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: b5173de >2014-04-29 09:19:05,217 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetVmsInfoVDSCommand] (ajp-/127.0.0.1:8702-7) FINISH, GetVmsInfoVDSCommand, log id: 71bce9be >2014-04-29 09:19:22,005 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.DisconnectStorageServerVDSCommand] (pool-4-thread-45) [1019a386] FINISH, DisconnectStorageServerVDSCommand, return: {1c52ac83-65c6-49de-9380-1f13ba3e4cc3=0}, log id: b5173de >2014-04-29 09:19:22,006 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [24b2854] Detach storage domain: after disconnect storage >2014-04-29 09:19:22,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResetISOPathVDSCommand] (pool-4-thread-46) [24b2854] START, ResetISOPathVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false), log id: 36df0f2e >2014-04-29 09:19:22,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ResetISOPathVDSCommand] (pool-4-thread-46) [24b2854] FINISH, ResetISOPathVDSCommand, log id: 36df0f2e >2014-04-29 09:19:22,009 INFO [org.ovirt.engine.core.bll.storage.DetachStorageDomainFromPoolCommand] (pool-4-thread-46) [24b2854] End detach storage domain >2014-04-29 09:19:22,013 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-46) [24b2854] Correlation ID: 24b2854, Job ID: d2ff0ea5-f39f-42a5-ad29-7532a444f5ab, Call Stack: null, Custom Event ID: -1, Message: Storage Domain iso was detached from Data Center NFS by admin@internal >2014-04-29 09:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-41) Autorecovering 1 hosts >2014-04-29 09:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-41) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7f5f9945 >2014-04-29 09:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] FINISH, SetVdsStatusVDSCommand, log id: 7f5f9945 >2014-04-29 09:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 8684504 >2014-04-29 09:20:00,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5a127dd0 >2014-04-29 09:20:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] Failed in GetHardwareInfoVDS method >2014-04-29 09:20:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:20:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@740210e2 >2014-04-29 09:20:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] HostName = 10.34.27.209 >2014-04-29 09:20:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:20:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [37b5c599] FINISH, GetHardwareInfoVDSCommand, log id: 5a127dd0 >2014-04-29 09:20:00,703 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [37b5c599] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:20:00,712 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-41) [7255347e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:20:00,713 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:20:00,713 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 8684504 >2014-04-29 09:20:00,728 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:20:00,729 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:20:02,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [1d42bd37] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5a48b547 >2014-04-29 09:20:07,574 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [1d42bd37] Failed in GetHardwareInfoVDS method >2014-04-29 09:20:07,575 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [1d42bd37] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:20:07,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [1d42bd37] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@12b88c7c >2014-04-29 09:20:07,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [1d42bd37] HostName = 10.34.27.209 >2014-04-29 09:20:07,575 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [1d42bd37] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:20:07,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [1d42bd37] FINISH, GetHardwareInfoVDSCommand, log id: 5a48b547 >2014-04-29 09:20:07,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [1d42bd37] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:20:07,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [1d42bd37] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:20:07,834 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-55) [75ac6f96] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:20:07,836 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [26da8bd9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:20:07,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [26da8bd9] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 34406b9a >2014-04-29 09:20:38,193 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [37c6b1d3] Lock Acquired to object EngineLock [exclusiveLocks= key: e4395463-7e5d-4c77-9dba-8e2edf383ef8 value: STORAGE >, sharedLocks= ] >2014-04-29 09:20:38,210 WARN [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [37c6b1d3] CanDoAction of action DeactivateStorageDomain failed. Reasons:VAR__TYPE__STORAGE__DOMAIN,VAR__ACTION__DEACTIVATE,ERROR_CANNOT_DEACTIVATE_DOMAIN_WITH_TASKS >2014-04-29 09:20:38,210 INFO [org.ovirt.engine.core.bll.storage.DeactivateStorageDomainCommand] (ajp-/127.0.0.1:8702-7) [37c6b1d3] Lock freed to object EngineLock [exclusiveLocks= key: e4395463-7e5d-4c77-9dba-8e2edf383ef8 value: STORAGE >, sharedLocks= ] >2014-04-29 09:22:14,778 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) SPMAsyncTask::PollTask: Polling task c5b5d4d0-5d7b-438a-849b-df35e9ed50ac (Parent Command ImportVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) returned status finished, result 'success'. >2014-04-29 09:22:14,782 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (DefaultQuartzScheduler_Worker-49) BaseAsyncTask::OnTaskEndSuccess: Task c5b5d4d0-5d7b-438a-849b-df35e9ed50ac (Parent Command ImportVmTemplate, Parameters Type org.ovirt.engine.core.common.asynctasks.AsyncTaskParameters) ended successfully. >2014-04-29 09:22:14,782 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-49) CommandAsyncTask::EndActionIfNecessary: All tasks of command af550ca5-d89b-477e-9825-9eeaaa24bf2c has ended -> executing EndAction >2014-04-29 09:22:14,782 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (DefaultQuartzScheduler_Worker-49) CommandAsyncTask::EndAction: Ending action for 1 tasks (command ID: af550ca5-d89b-477e-9825-9eeaaa24bf2c): calling EndAction . >2014-04-29 09:22:14,782 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::EndCommandAction [within thread] context: Attempting to EndAction ImportVmTemplate, executionIndex: 0 >2014-04-29 09:22:14,788 INFO [org.ovirt.engine.core.bll.ImportVmTemplateCommand] (pool-4-thread-50) Ending command successfully: org.ovirt.engine.core.bll.ImportVmTemplateCommand >2014-04-29 09:22:14,795 INFO [org.ovirt.engine.core.bll.CopyImageGroupCommand] (pool-4-thread-50) [17c0ce43] Ending command successfully: org.ovirt.engine.core.bll.CopyImageGroupCommand >2014-04-29 09:22:14,796 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [17c0ce43] START, GetImageInfoVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = 522919d2-161c-403b-ba40-b670fa42994b, imageGroupId = 43f667af-cd78-4e40-b5a9-004bd91bb406, imageId = dfde1f5e-cd6f-44ba-8af5-84d52055a756), log id: 16e48bcc >2014-04-29 09:22:15,068 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.GetImageInfoVDSCommand] (pool-4-thread-50) [17c0ce43] FINISH, GetImageInfoVDSCommand, return: org.ovirt.engine.core.common.businessentities.DiskImage@79efd934, log id: 16e48bcc >2014-04-29 09:22:15,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (pool-4-thread-50) Correlation ID: 5199bc16, Job ID: cc2fa90c-b93e-4c16-a590-e821b06c0ca0, Call Stack: null, Custom Event ID: -1, Message: Template sswe_prd_v03.0.0 was imported successfully to Data Center NFS, Cluster NFS >2014-04-29 09:22:15,112 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type ImportVmTemplate completed, handling the result. >2014-04-29 09:22:15,112 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::HandleEndActionResult [within thread]: EndAction for action type ImportVmTemplate succeeded, clearing tasks. >2014-04-29 09:22:15,140 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) SPMAsyncTask::ClearAsyncTask: Attempting to clear task c5b5d4d0-5d7b-438a-849b-df35e9ed50ac >2014-04-29 09:22:15,140 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) START, SPMClearTaskVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, taskId = c5b5d4d0-5d7b-438a-849b-df35e9ed50ac), log id: 13c1108a >2014-04-29 09:22:15,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) START, HSMClearTaskVDSCommand(HostName = 10.34.1.251, HostId = 56f84250-7f1e-46c8-924f-66c454fd877a, taskId=c5b5d4d0-5d7b-438a-849b-df35e9ed50ac), log id: 6824de27 >2014-04-29 09:22:15,433 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.HSMClearTaskVDSCommand] (pool-4-thread-50) FINISH, HSMClearTaskVDSCommand, log id: 6824de27 >2014-04-29 09:22:15,433 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.SPMClearTaskVDSCommand] (pool-4-thread-50) FINISH, SPMClearTaskVDSCommand, log id: 13c1108a >2014-04-29 09:22:15,437 INFO [org.ovirt.engine.core.bll.SPMAsyncTask] (pool-4-thread-50) BaseAsyncTask::RemoveTaskFromDB: Removed task c5b5d4d0-5d7b-438a-849b-df35e9ed50ac from DataBase >2014-04-29 09:22:15,437 INFO [org.ovirt.engine.core.bll.CommandAsyncTask] (pool-4-thread-50) CommandAsyncTask::HandleEndActionResult [within thread]: Removing CommandMultiAsyncTasks object for entity af550ca5-d89b-477e-9825-9eeaaa24bf2c >2014-04-29 09:23:07,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [26da8bd9] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 09:23:07,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [26da8bd9] FINISH, ConnectStorageServerVDSCommand, log id: 34406b9a >2014-04-29 09:23:07,848 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [26da8bd9] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 09:23:07,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 215027de >2014-04-29 09:23:08,281 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 215027de >2014-04-29 09:23:08,281 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:23:08,290 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-55) [b7dd243] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:23:08,291 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [b7dd243] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: a7d9179 >2014-04-29 09:23:08,294 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [b7dd243] FINISH, SetVdsStatusVDSCommand, log id: a7d9179 >2014-04-29 09:23:08,300 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [b7dd243] Correlation ID: b7dd243, Job ID: 6af2c774-fd6a-469c-ad24-588dcf62e5dc, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:23:08,305 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [b7dd243] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:23:08,307 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [b7dd243] Correlation ID: 75ac6f96, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:23:08,310 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [1899fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:23:08,312 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [67c34fbf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:23:08,312 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [67c34fbf] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 09:23:35,449 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-49) Setting new tasks map. The map contains now 0 tasks >2014-04-29 09:23:35,449 INFO [org.ovirt.engine.core.bll.AsyncTaskManager] (DefaultQuartzScheduler_Worker-49) Cleared all tasks of pool f03e56af-56f8-4150-9517-271bec2305d9. >2014-04-29 09:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-84) Autorecovering 1 hosts >2014-04-29 09:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-84) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 112f23 >2014-04-29 09:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] FINISH, SetVdsStatusVDSCommand, log id: 112f23 >2014-04-29 09:25:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4163c95e >2014-04-29 09:25:00,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 77f71e00 >2014-04-29 09:25:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] Failed in GetHardwareInfoVDS method >2014-04-29 09:25:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:25:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@d2cf88 >2014-04-29 09:25:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] HostName = 10.34.27.209 >2014-04-29 09:25:00,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:25:00,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [5da68c3c] FINISH, GetHardwareInfoVDSCommand, log id: 77f71e00 >2014-04-29 09:25:00,694 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [5da68c3c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:25:00,714 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [28510b04] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:25:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [2e21be14] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:25:00,716 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [2e21be14] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4163c95e >2014-04-29 09:25:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [2e21be14] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:25:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [2e21be14] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:25:02,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6642803d >2014-04-29 09:25:02,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) Failed in GetHardwareInfoVDS method >2014-04-29 09:25:02,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:25:02,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@17f1593e >2014-04-29 09:25:02,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) HostName = 10.34.27.209 >2014-04-29 09:25:02,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:25:02,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) FINISH, GetHardwareInfoVDSCommand, log id: 6642803d >2014-04-29 09:25:02,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:25:02,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:25:03,111 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-45) [370e528c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:25:03,113 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [4d325dfe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:25:03,120 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [4d325dfe] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 127dfbe9 >2014-04-29 09:28:03,122 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [4d325dfe] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 09:28:03,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [4d325dfe] FINISH, ConnectStorageServerVDSCommand, log id: 127dfbe9 >2014-04-29 09:28:03,122 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [4d325dfe] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 09:28:03,128 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3cbcbad4 >2014-04-29 09:28:03,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3cbcbad4 >2014-04-29 09:28:03,566 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:28:03,610 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-45) [b780416] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:28:03,612 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [b780416] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 35b6d304 >2014-04-29 09:28:03,633 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [b780416] FINISH, SetVdsStatusVDSCommand, log id: 35b6d304 >2014-04-29 09:28:03,652 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [b780416] Correlation ID: b780416, Job ID: 8fd3ceeb-da30-43c7-b574-8791fb6c1711, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:28:03,661 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [b780416] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:28:03,663 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [b780416] Correlation ID: 370e528c, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:28:03,669 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [c09d4c1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:28:03,670 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [790d05f6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:28:03,671 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) [790d05f6] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 09:30:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) Autorecovering 1 hosts >2014-04-29 09:30:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:30:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:30:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:30:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:30:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:30:00,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5037d953 >2014-04-29 09:30:00,021 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] FINISH, SetVdsStatusVDSCommand, log id: 5037d953 >2014-04-29 09:30:00,022 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 19586043 >2014-04-29 09:30:00,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1485d571 >2014-04-29 09:30:00,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] Failed in GetHardwareInfoVDS method >2014-04-29 09:30:00,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:30:00,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@34e6390 >2014-04-29 09:30:00,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] HostName = 10.34.27.209 >2014-04-29 09:30:00,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:30:00,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [265c598d] FINISH, GetHardwareInfoVDSCommand, log id: 1485d571 >2014-04-29 09:30:00,730 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [265c598d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:30:00,747 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-97) [6d8ea566] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:30:00,749 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-97) [688583cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:30:00,750 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) [688583cd] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 19586043 >2014-04-29 09:30:00,766 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [688583cd] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:30:00,767 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [688583cd] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:30:04,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 321f77b0 >2014-04-29 09:30:04,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) Failed in GetHardwareInfoVDS method >2014-04-29 09:30:04,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:30:04,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@28e4c540 >2014-04-29 09:30:04,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) HostName = 10.34.27.209 >2014-04-29 09:30:04,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:30:04,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) FINISH, GetHardwareInfoVDSCommand, log id: 321f77b0 >2014-04-29 09:30:04,259 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:30:04,270 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:30:04,489 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-95) [21b1ebcc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:30:04,491 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [37be9bb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:30:04,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [37be9bb] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 27f9e36c >2014-04-29 09:33:04,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [37be9bb] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 09:33:04,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [37be9bb] FINISH, ConnectStorageServerVDSCommand, log id: 27f9e36c >2014-04-29 09:33:04,504 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [37be9bb] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 09:33:04,509 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 560cbd9c >2014-04-29 09:33:04,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 560cbd9c >2014-04-29 09:33:04,951 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:33:04,963 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-95) [45bde7b4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:33:04,964 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [45bde7b4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 50501648 >2014-04-29 09:33:04,967 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [45bde7b4] FINISH, SetVdsStatusVDSCommand, log id: 50501648 >2014-04-29 09:33:04,974 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [45bde7b4] Correlation ID: 45bde7b4, Job ID: 1155d806-db11-4eea-b375-6ec53e70b3ca, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:33:04,981 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [45bde7b4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:33:04,984 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [45bde7b4] Correlation ID: 21b1ebcc, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:33:04,988 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [4bfc5610] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:33:04,990 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [6798afa9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:33:04,991 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [6798afa9] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 09:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) Autorecovering 1 hosts >2014-04-29 09:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2c9a78c6 >2014-04-29 09:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] FINISH, SetVdsStatusVDSCommand, log id: 2c9a78c6 >2014-04-29 09:35:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 66e22f40 >2014-04-29 09:35:00,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2e465428 >2014-04-29 09:35:01,047 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] Failed in GetHardwareInfoVDS method >2014-04-29 09:35:01,048 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:35:01,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@b3f25d4 >2014-04-29 09:35:01,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] HostName = 10.34.27.209 >2014-04-29 09:35:01,048 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:35:01,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [6bc31a56] FINISH, GetHardwareInfoVDSCommand, log id: 2e465428 >2014-04-29 09:35:01,058 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [6bc31a56] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:35:01,071 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-15) [241cb65d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:35:01,072 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-15) [57935dec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:35:01,073 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) [57935dec] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 66e22f40 >2014-04-29 09:35:01,091 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [57935dec] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:35:01,092 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [57935dec] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:35:02,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3229324e >2014-04-29 09:35:02,585 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] Failed in GetHardwareInfoVDS method >2014-04-29 09:35:02,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:35:02,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2ea5b89c >2014-04-29 09:35:02,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] HostName = 10.34.27.209 >2014-04-29 09:35:02,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:35:02,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [6906c39a] FINISH, GetHardwareInfoVDSCommand, log id: 3229324e >2014-04-29 09:35:02,593 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [6906c39a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:35:02,602 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [6906c39a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:35:02,823 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-18) [194b33d5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:35:02,825 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-18) [108d8150] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:35:02,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [108d8150] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 474de924 >2014-04-29 09:36:04,979 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-71) Start refreshing all users data >2014-04-29 09:36:06,618 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VMs/Templates Ovf. >2014-04-29 09:36:06,619 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update VM OVFs in Data Center NFS >2014-04-29 09:36:06,624 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated VM OVFs in Data Center NFS >2014-04-29 09:36:06,624 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to update template OVFs in Data Center NFS >2014-04-29 09:36:06,645 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (DefaultQuartzScheduler_Worker-38) START, UpdateVMVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = 00000000-0000-0000-0000-000000000000, infoDictionary.size = 1), log id: 37cea871 >2014-04-29 09:36:06,928 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.UpdateVMVDSCommand] (DefaultQuartzScheduler_Worker-38) FINISH, UpdateVMVDSCommand, log id: 37cea871 >2014-04-29 09:36:06,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully updated templates OVFs in Data Center NFS >2014-04-29 09:36:06,937 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 09:36:06,939 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.RemoveVMVDSCommand] (DefaultQuartzScheduler_Worker-38) START, RemoveVMVDSCommand( storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, ignoreFailoverLimit = false, storageDomainId = 00000000-0000-0000-0000-000000000000, vmGuid = b066ff4b-970f-4305-94a4-1f32f7fa6fe2), log id: 7d27dbba >2014-04-29 09:36:07,207 INFO [org.ovirt.engine.core.vdsbroker.irsbroker.RemoveVMVDSCommand] (DefaultQuartzScheduler_Worker-38) FINISH, RemoveVMVDSCommand, log id: 7d27dbba >2014-04-29 09:36:07,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-38) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 09:38:02,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [108d8150] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 09:38:02,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [108d8150] FINISH, ConnectStorageServerVDSCommand, log id: 474de924 >2014-04-29 09:38:02,836 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-18) [108d8150] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 09:38:02,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 17338285 >2014-04-29 09:38:03,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 17338285 >2014-04-29 09:38:03,275 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:38:03,289 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-18) [536b4c9e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:38:03,290 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [536b4c9e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: cb594ad >2014-04-29 09:38:03,292 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [536b4c9e] FINISH, SetVdsStatusVDSCommand, log id: cb594ad >2014-04-29 09:38:03,298 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [536b4c9e] Correlation ID: 536b4c9e, Job ID: 8a8c71c8-fe41-4b1e-87d4-81d42019aafa, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:38:03,308 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [536b4c9e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:38:03,310 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [536b4c9e] Correlation ID: 194b33d5, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:38:03,313 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-18) [535c520] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:38:03,314 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-18) [213931a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:38:03,315 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) [213931a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 09:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-76) [3f83e0d2] Autorecovering 1 hosts >2014-04-29 09:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-76) [3f83e0d2] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:40:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 69bcd50a >2014-04-29 09:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] FINISH, SetVdsStatusVDSCommand, log id: 69bcd50a >2014-04-29 09:40:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 445d4790 >2014-04-29 09:40:00,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4a77c46d >2014-04-29 09:40:00,962 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] Failed in GetHardwareInfoVDS method >2014-04-29 09:40:00,962 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:40:00,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7d00f64c >2014-04-29 09:40:00,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] HostName = 10.34.27.209 >2014-04-29 09:40:00,962 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:40:00,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [15a3bf10] FINISH, GetHardwareInfoVDSCommand, log id: 4a77c46d >2014-04-29 09:40:00,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [15a3bf10] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:40:00,976 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-76) [562ef069] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:40:00,979 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:40:00,979 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 445d4790 >2014-04-29 09:40:00,993 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:40:00,993 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:40:03,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 319f94c3 >2014-04-29 09:40:03,913 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) Failed in GetHardwareInfoVDS method >2014-04-29 09:40:03,913 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:40:03,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6c7b867c >2014-04-29 09:40:03,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) HostName = 10.34.27.209 >2014-04-29 09:40:03,913 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:40:03,913 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) FINISH, GetHardwareInfoVDSCommand, log id: 319f94c3 >2014-04-29 09:40:03,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:40:03,927 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:40:04,143 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-80) [31b7cf2f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:40:04,144 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [1044a808] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:40:04,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [1044a808] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 639b5722 >2014-04-29 09:43:04,152 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [1044a808] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 09:43:04,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [1044a808] FINISH, ConnectStorageServerVDSCommand, log id: 639b5722 >2014-04-29 09:43:04,152 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [1044a808] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 09:43:04,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 48e5cc30 >2014-04-29 09:43:04,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 48e5cc30 >2014-04-29 09:43:04,582 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:43:04,594 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-80) [3c7e9189] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:43:04,595 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [3c7e9189] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 668a37d >2014-04-29 09:43:04,597 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [3c7e9189] FINISH, SetVdsStatusVDSCommand, log id: 668a37d >2014-04-29 09:43:04,604 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [3c7e9189] Correlation ID: 3c7e9189, Job ID: f9f90189-7624-4dac-83bf-5ffb6d2e0f0e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:43:04,610 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [3c7e9189] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:43:04,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [3c7e9189] Correlation ID: 31b7cf2f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:43:04,622 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-80) [25305f1f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:43:04,623 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-80) [365e941f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:43:04,624 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) [365e941f] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 09:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-74) Autorecovering 1 hosts >2014-04-29 09:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-74) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2732e8c9 >2014-04-29 09:45:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] FINISH, SetVdsStatusVDSCommand, log id: 2732e8c9 >2014-04-29 09:45:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4a5ab76b >2014-04-29 09:45:00,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 78eec08 >2014-04-29 09:45:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] Failed in GetHardwareInfoVDS method >2014-04-29 09:45:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:45:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@382a0518 >2014-04-29 09:45:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] HostName = 10.34.27.209 >2014-04-29 09:45:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:45:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6386329f] FINISH, GetHardwareInfoVDSCommand, log id: 78eec08 >2014-04-29 09:45:00,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [6386329f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:45:00,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-74) [4017f715] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:45:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:45:00,710 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4a5ab76b >2014-04-29 09:45:00,725 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:45:00,726 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:45:01,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 50edbb40 >2014-04-29 09:45:02,198 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) Failed in GetHardwareInfoVDS method >2014-04-29 09:45:02,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:45:02,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@27a09042 >2014-04-29 09:45:02,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) HostName = 10.34.27.209 >2014-04-29 09:45:02,199 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:45:02,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) FINISH, GetHardwareInfoVDSCommand, log id: 50edbb40 >2014-04-29 09:45:02,203 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:45:02,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:45:02,441 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-59) [74f293c0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:45:02,443 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-59) [22e19ce9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:45:02,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [22e19ce9] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 792e4832 >2014-04-29 09:48:02,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [22e19ce9] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 09:48:02,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [22e19ce9] FINISH, ConnectStorageServerVDSCommand, log id: 792e4832 >2014-04-29 09:48:02,455 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-59) [22e19ce9] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 09:48:02,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 16d8c8b0 >2014-04-29 09:48:02,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 16d8c8b0 >2014-04-29 09:48:02,903 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:48:02,914 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-59) [3fc2add2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:48:02,915 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [3fc2add2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7adecd20 >2014-04-29 09:48:02,918 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [3fc2add2] FINISH, SetVdsStatusVDSCommand, log id: 7adecd20 >2014-04-29 09:48:02,926 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [3fc2add2] Correlation ID: 3fc2add2, Job ID: c59be3c4-b805-4653-93c1-dfe803719024, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:48:02,932 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [3fc2add2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:48:02,934 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [3fc2add2] Correlation ID: 74f293c0, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:48:02,938 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-59) [61c173cd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:48:02,942 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-59) [19f188f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:48:02,943 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) [19f188f3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 09:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-69) Autorecovering 1 hosts >2014-04-29 09:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-69) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2df554db >2014-04-29 09:50:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] FINISH, SetVdsStatusVDSCommand, log id: 2df554db >2014-04-29 09:50:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7c5eeaa2 >2014-04-29 09:50:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 35938758 >2014-04-29 09:50:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] Failed in GetHardwareInfoVDS method >2014-04-29 09:50:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:50:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@689abde2 >2014-04-29 09:50:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] HostName = 10.34.27.209 >2014-04-29 09:50:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:50:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] FINISH, GetHardwareInfoVDSCommand, log id: 35938758 >2014-04-29 09:50:00,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [2fac6e0b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:50:00,711 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-69) [4ca35692] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:50:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-69) [273b756f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:50:00,717 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-69) [273b756f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7c5eeaa2 >2014-04-29 09:50:00,738 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [273b756f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:50:00,739 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [273b756f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:50:03,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3a0e669a >2014-04-29 09:50:03,522 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) Failed in GetHardwareInfoVDS method >2014-04-29 09:50:03,522 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:50:03,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@772263d >2014-04-29 09:50:03,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) HostName = 10.34.27.209 >2014-04-29 09:50:03,522 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:50:03,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) FINISH, GetHardwareInfoVDSCommand, log id: 3a0e669a >2014-04-29 09:50:03,528 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:50:03,539 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:50:03,755 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-65) [60fc4f39] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:50:03,757 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-65) [2f817fe5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:50:03,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-65) [2f817fe5] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5e0e504c >2014-04-29 09:53:03,775 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-65) [2f817fe5] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 09:53:03,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-65) [2f817fe5] FINISH, ConnectStorageServerVDSCommand, log id: 5e0e504c >2014-04-29 09:53:03,775 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-65) [2f817fe5] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 09:53:03,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 25b7b589 >2014-04-29 09:53:04,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 25b7b589 >2014-04-29 09:53:04,216 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:53:04,231 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-65) [49cc807b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:53:04,232 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [49cc807b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 55f3b597 >2014-04-29 09:53:04,235 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [49cc807b] FINISH, SetVdsStatusVDSCommand, log id: 55f3b597 >2014-04-29 09:53:04,241 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [49cc807b] Correlation ID: 49cc807b, Job ID: bbcede2e-96bd-465c-aad0-dcf4229b891c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:53:04,248 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [49cc807b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:53:04,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [49cc807b] Correlation ID: 60fc4f39, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:53:04,256 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-65) [6db46ef6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:53:04,257 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-65) [570caa35] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:53:04,258 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) [570caa35] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 09:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-49) Autorecovering 1 hosts >2014-04-29 09:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-49) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 09:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 09:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 09:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2bb1f0cb >2014-04-29 09:55:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] FINISH, SetVdsStatusVDSCommand, log id: 2bb1f0cb >2014-04-29 09:55:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 342c0460 >2014-04-29 09:55:00,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 72ef7b8d >2014-04-29 09:55:00,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] Failed in GetHardwareInfoVDS method >2014-04-29 09:55:00,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:55:00,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2d91ecd7 >2014-04-29 09:55:00,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] HostName = 10.34.27.209 >2014-04-29 09:55:00,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:55:00,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [127947d6] FINISH, GetHardwareInfoVDSCommand, log id: 72ef7b8d >2014-04-29 09:55:00,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [127947d6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:55:00,807 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-49) [6f00c43d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:55:00,808 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:55:00,809 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 342c0460 >2014-04-29 09:55:00,829 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 09:55:00,830 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 09:55:01,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 39ba363d >2014-04-29 09:55:01,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) Failed in GetHardwareInfoVDS method >2014-04-29 09:55:01,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:55:01,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7f1bd5b4 >2014-04-29 09:55:01,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) HostName = 10.34.27.209 >2014-04-29 09:55:01,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 09:55:01,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) FINISH, GetHardwareInfoVDSCommand, log id: 39ba363d >2014-04-29 09:55:01,889 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 09:55:01,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 09:55:02,144 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-44) [1995d3f3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:55:02,148 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [8eec15f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 09:55:02,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [8eec15f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7d0b0f8d >2014-04-29 09:58:02,155 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [8eec15f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 09:58:02,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [8eec15f] FINISH, ConnectStorageServerVDSCommand, log id: 7d0b0f8d >2014-04-29 09:58:02,155 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [8eec15f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 09:58:02,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1a69f308 >2014-04-29 09:58:02,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 1a69f308 >2014-04-29 09:58:02,602 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 09:58:02,612 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-44) [7e999bc1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:58:02,613 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [7e999bc1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 56a9d241 >2014-04-29 09:58:02,616 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [7e999bc1] FINISH, SetVdsStatusVDSCommand, log id: 56a9d241 >2014-04-29 09:58:02,622 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [7e999bc1] Correlation ID: 7e999bc1, Job ID: 854c318a-41c8-4993-ae26-a98e7bdb989a, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 09:58:02,627 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [7e999bc1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 09:58:02,631 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [7e999bc1] Correlation ID: 1995d3f3, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 09:58:02,636 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [67d05d59] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:58:02,639 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [2619bd68] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 09:58:02,639 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) [2619bd68] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [57935dec] Autorecovering 1 hosts >2014-04-29 10:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [57935dec] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 63f8a495 >2014-04-29 10:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] FINISH, SetVdsStatusVDSCommand, log id: 63f8a495 >2014-04-29 10:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 44117c3 >2014-04-29 10:00:00,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5dffc3f0 >2014-04-29 10:00:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] Failed in GetHardwareInfoVDS method >2014-04-29 10:00:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:00:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4298f1c6 >2014-04-29 10:00:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] HostName = 10.34.27.209 >2014-04-29 10:00:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:00:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [4c904acb] FINISH, GetHardwareInfoVDSCommand, log id: 5dffc3f0 >2014-04-29 10:00:00,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [4c904acb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:00:00,710 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-15) [4aca30a6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:00:00,712 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:00:00,713 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 44117c3 >2014-04-29 10:00:00,727 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:00:00,728 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:00:03,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2c8a94c6 >2014-04-29 10:00:03,219 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) Failed in GetHardwareInfoVDS method >2014-04-29 10:00:03,219 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:00:03,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3e2d6a02 >2014-04-29 10:00:03,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) HostName = 10.34.27.209 >2014-04-29 10:00:03,219 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:00:03,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) FINISH, GetHardwareInfoVDSCommand, log id: 2c8a94c6 >2014-04-29 10:00:03,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:00:03,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:00:03,456 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-66) [3622f5ed] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:00:03,458 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [7015f346] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:00:03,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [7015f346] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5a034c1d >2014-04-29 10:03:03,468 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [7015f346] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:03:03,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [7015f346] FINISH, ConnectStorageServerVDSCommand, log id: 5a034c1d >2014-04-29 10:03:03,468 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [7015f346] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:03:03,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4e3efa98 >2014-04-29 10:03:03,920 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 4e3efa98 >2014-04-29 10:03:03,920 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:03:03,930 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-66) [b6f72fb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:03:03,931 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [b6f72fb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 36a95d55 >2014-04-29 10:03:03,933 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [b6f72fb] FINISH, SetVdsStatusVDSCommand, log id: 36a95d55 >2014-04-29 10:03:03,940 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [b6f72fb] Correlation ID: b6f72fb, Job ID: 56ba0cf7-c361-4c93-bc73-778412708d27, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:03:03,946 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [b6f72fb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:03:03,948 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [b6f72fb] Correlation ID: 3622f5ed, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:03:03,959 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-66) [1243a0b7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:03:03,960 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-66) [4a646c36] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:03:03,961 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [4a646c36] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-21) Autorecovering 1 hosts >2014-04-29 10:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-21) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4a4981d1 >2014-04-29 10:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] FINISH, SetVdsStatusVDSCommand, log id: 4a4981d1 >2014-04-29 10:05:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1476d73b >2014-04-29 10:05:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 845ee3f >2014-04-29 10:05:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] Failed in GetHardwareInfoVDS method >2014-04-29 10:05:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:05:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@271c6e57 >2014-04-29 10:05:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] HostName = 10.34.27.209 >2014-04-29 10:05:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:05:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] FINISH, GetHardwareInfoVDSCommand, log id: 845ee3f >2014-04-29 10:05:00,710 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [5dd65c8e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:05:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-21) [21035317] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:05:00,726 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-21) [16d7e943] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:05:00,727 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) [16d7e943] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1476d73b >2014-04-29 10:05:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [16d7e943] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:05:00,743 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [16d7e943] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:05:01,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67c34fbf] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 8021291 >2014-04-29 10:05:01,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67c34fbf] Failed in GetHardwareInfoVDS method >2014-04-29 10:05:01,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67c34fbf] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:05:01,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67c34fbf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@71294f17 >2014-04-29 10:05:01,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67c34fbf] HostName = 10.34.27.209 >2014-04-29 10:05:01,535 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67c34fbf] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:05:01,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67c34fbf] FINISH, GetHardwareInfoVDSCommand, log id: 8021291 >2014-04-29 10:05:01,540 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [67c34fbf] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:05:01,550 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [67c34fbf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:05:01,769 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-55) [7c947162] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:05:01,770 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [7cea64d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:05:01,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [7cea64d3] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5d0199b7 >2014-04-29 10:08:01,778 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [7cea64d3] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:08:01,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [7cea64d3] FINISH, ConnectStorageServerVDSCommand, log id: 5d0199b7 >2014-04-29 10:08:01,778 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [7cea64d3] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:08:01,781 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 64be502a >2014-04-29 10:08:02,214 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 64be502a >2014-04-29 10:08:02,214 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:08:02,225 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-55) [590ca63] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:08:02,226 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [590ca63] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 13049856 >2014-04-29 10:08:02,228 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [590ca63] FINISH, SetVdsStatusVDSCommand, log id: 13049856 >2014-04-29 10:08:02,237 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [590ca63] Correlation ID: 590ca63, Job ID: 27057c04-2370-4d5f-852a-7201992eda02, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:08:02,243 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [590ca63] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:08:02,245 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [590ca63] Correlation ID: 7c947162, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:08:02,249 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [e3ebbde] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:08:02,253 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:08:02,254 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [570caa35] Autorecovering 1 hosts >2014-04-29 10:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [570caa35] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:10:00,021 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4fa64f4c >2014-04-29 10:10:00,027 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] FINISH, SetVdsStatusVDSCommand, log id: 4fa64f4c >2014-04-29 10:10:00,029 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7965b757 >2014-04-29 10:10:00,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1048e20c >2014-04-29 10:10:00,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] Failed in GetHardwareInfoVDS method >2014-04-29 10:10:00,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:10:00,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@ccf72 >2014-04-29 10:10:00,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] HostName = 10.34.27.209 >2014-04-29 10:10:00,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:10:00,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [35859e3b] FINISH, GetHardwareInfoVDSCommand, log id: 1048e20c >2014-04-29 10:10:00,728 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [35859e3b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:10:00,736 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-65) [659ede8a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:10:00,743 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-65) [68acb3c9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:10:00,744 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) [68acb3c9] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7965b757 >2014-04-29 10:10:00,759 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [68acb3c9] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:10:00,760 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [68acb3c9] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:10:02,647 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 34b316b4 >2014-04-29 10:10:02,864 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) Failed in GetHardwareInfoVDS method >2014-04-29 10:10:02,864 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:10:02,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@59979e73 >2014-04-29 10:10:02,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) HostName = 10.34.27.209 >2014-04-29 10:10:02,864 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:10:02,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) FINISH, GetHardwareInfoVDSCommand, log id: 34b316b4 >2014-04-29 10:10:02,869 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:10:02,883 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:10:03,099 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-92) [13b8d2f0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:10:03,101 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-92) [68909f9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:10:03,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [68909f9d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 534ee76d >2014-04-29 10:13:03,108 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [68909f9d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:13:03,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [68909f9d] FINISH, ConnectStorageServerVDSCommand, log id: 534ee76d >2014-04-29 10:13:03,108 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-92) [68909f9d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:13:03,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 39a278df >2014-04-29 10:13:03,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 39a278df >2014-04-29 10:13:03,542 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:13:03,556 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-92) [10007244] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:13:03,557 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [10007244] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2275f74b >2014-04-29 10:13:03,559 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [10007244] FINISH, SetVdsStatusVDSCommand, log id: 2275f74b >2014-04-29 10:13:03,565 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [10007244] Correlation ID: 10007244, Job ID: 49a59f70-a96c-4c2c-b7f2-6f63c68a9dbc, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:13:03,573 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [10007244] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:13:03,575 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [10007244] Correlation ID: 13b8d2f0, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:13:03,579 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-92) [89095ca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:13:03,580 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-92) [5e7dac47] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:13:03,581 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [5e7dac47] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-64) [2fd0bbdc] Autorecovering 1 hosts >2014-04-29 10:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-64) [2fd0bbdc] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:15:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7e2de3e2 >2014-04-29 10:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] FINISH, SetVdsStatusVDSCommand, log id: 7e2de3e2 >2014-04-29 10:15:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: da7c379 >2014-04-29 10:15:00,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 324e89c9 >2014-04-29 10:15:00,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] Failed in GetHardwareInfoVDS method >2014-04-29 10:15:00,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:15:00,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@423804ec >2014-04-29 10:15:00,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] HostName = 10.34.27.209 >2014-04-29 10:15:00,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:15:00,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [103a65b] FINISH, GetHardwareInfoVDSCommand, log id: 324e89c9 >2014-04-29 10:15:00,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [103a65b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:15:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [1d81039c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:15:00,711 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:15:00,711 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: da7c379 >2014-04-29 10:15:00,726 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:15:00,727 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:15:03,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 54d09e0d >2014-04-29 10:15:04,162 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) Failed in GetHardwareInfoVDS method >2014-04-29 10:15:04,162 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:15:04,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4c3892fb >2014-04-29 10:15:04,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) HostName = 10.34.27.209 >2014-04-29 10:15:04,162 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:15:04,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) FINISH, GetHardwareInfoVDSCommand, log id: 54d09e0d >2014-04-29 10:15:04,168 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:15:04,179 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:15:04,395 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-29) [46811b30] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:15:04,397 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [4544ad7f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:15:04,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [4544ad7f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3a44f034 >2014-04-29 10:18:04,404 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [4544ad7f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:18:04,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [4544ad7f] FINISH, ConnectStorageServerVDSCommand, log id: 3a44f034 >2014-04-29 10:18:04,405 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [4544ad7f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:18:04,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 26780b12 >2014-04-29 10:18:04,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 26780b12 >2014-04-29 10:18:04,855 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:18:04,866 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-29) [700e73a3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:18:04,867 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [700e73a3] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 63f3ea1 >2014-04-29 10:18:04,869 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [700e73a3] FINISH, SetVdsStatusVDSCommand, log id: 63f3ea1 >2014-04-29 10:18:04,875 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [700e73a3] Correlation ID: 700e73a3, Job ID: 3d038b99-a1de-49a2-8ddb-b5ca3819a1c9, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:18:04,881 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [700e73a3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:18:04,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [700e73a3] Correlation ID: 46811b30, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:18:04,888 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-29) [171fcb5b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:18:04,890 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-29) [38b1be3b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:18:04,891 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) [38b1be3b] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-99) Autorecovering 1 hosts >2014-04-29 10:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-99) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:20:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:20:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:20:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:20:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 55915b25 >2014-04-29 10:20:00,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] FINISH, SetVdsStatusVDSCommand, log id: 55915b25 >2014-04-29 10:20:00,019 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 68d3c5fa >2014-04-29 10:20:00,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 62362443 >2014-04-29 10:20:00,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] Failed in GetHardwareInfoVDS method >2014-04-29 10:20:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:20:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5086dd14 >2014-04-29 10:20:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] HostName = 10.34.27.209 >2014-04-29 10:20:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:20:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [60986b49] FINISH, GetHardwareInfoVDSCommand, log id: 62362443 >2014-04-29 10:20:00,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [60986b49] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:20:00,703 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-99) [1d78ffac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:20:00,704 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:20:00,705 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 68d3c5fa >2014-04-29 10:20:00,719 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:20:00,720 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:20:02,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7e397ccf >2014-04-29 10:20:02,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) Failed in GetHardwareInfoVDS method >2014-04-29 10:20:02,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:20:02,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@33b74d9d >2014-04-29 10:20:02,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) HostName = 10.34.27.209 >2014-04-29 10:20:02,458 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:20:02,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) FINISH, GetHardwareInfoVDSCommand, log id: 7e397ccf >2014-04-29 10:20:02,463 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:20:02,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:20:02,690 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-53) [28e4b903] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:20:02,692 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-53) [6d745667] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:20:02,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [6d745667] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5ea2020b >2014-04-29 10:23:02,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [6d745667] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:23:02,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [6d745667] FINISH, ConnectStorageServerVDSCommand, log id: 5ea2020b >2014-04-29 10:23:02,702 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-53) [6d745667] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:23:02,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 21613a9 >2014-04-29 10:23:03,156 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 21613a9 >2014-04-29 10:23:03,156 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:23:03,168 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-53) [59ee74b4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:23:03,169 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [59ee74b4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1b92c481 >2014-04-29 10:23:03,171 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [59ee74b4] FINISH, SetVdsStatusVDSCommand, log id: 1b92c481 >2014-04-29 10:23:03,178 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [59ee74b4] Correlation ID: 59ee74b4, Job ID: ec3105a7-5cac-4003-80df-1a7203ab168d, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:23:03,184 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [59ee74b4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:23:03,186 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [59ee74b4] Correlation ID: 28e4b903, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:23:03,189 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-53) [591c1c9d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:23:03,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:23:03,191 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-83) Autorecovering 1 hosts >2014-04-29 10:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-83) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4ab4cbe9 >2014-04-29 10:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] FINISH, SetVdsStatusVDSCommand, log id: 4ab4cbe9 >2014-04-29 10:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1a8c8d0 >2014-04-29 10:25:00,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 45f5a35c >2014-04-29 10:25:00,679 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] Failed in GetHardwareInfoVDS method >2014-04-29 10:25:00,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:25:00,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1b810e6 >2014-04-29 10:25:00,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] HostName = 10.34.27.209 >2014-04-29 10:25:00,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:25:00,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [5d23401e] FINISH, GetHardwareInfoVDSCommand, log id: 45f5a35c >2014-04-29 10:25:00,685 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [5d23401e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:25:00,698 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-83) [7aeeaa94] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:25:00,701 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:25:00,702 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1a8c8d0 >2014-04-29 10:25:00,724 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:25:00,725 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:25:03,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 61b09b09 >2014-04-29 10:25:03,776 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) Failed in GetHardwareInfoVDS method >2014-04-29 10:25:03,776 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:25:03,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@11929dfa >2014-04-29 10:25:03,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) HostName = 10.34.27.209 >2014-04-29 10:25:03,776 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:25:03,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) FINISH, GetHardwareInfoVDSCommand, log id: 61b09b09 >2014-04-29 10:25:03,780 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:25:03,788 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:25:04,006 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-96) [138ac13f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:25:04,007 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-96) [356a5971] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:25:04,013 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [356a5971] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3fb3fc32 >2014-04-29 10:28:04,015 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [356a5971] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:28:04,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [356a5971] FINISH, ConnectStorageServerVDSCommand, log id: 3fb3fc32 >2014-04-29 10:28:04,015 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-96) [356a5971] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:28:04,018 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4438912b >2014-04-29 10:28:04,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 4438912b >2014-04-29 10:28:04,457 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:28:04,469 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-96) [3d28f338] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:28:04,471 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-96) [3d28f338] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1d8b0935 >2014-04-29 10:28:04,480 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-96) [3d28f338] FINISH, SetVdsStatusVDSCommand, log id: 1d8b0935 >2014-04-29 10:28:04,487 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [3d28f338] Correlation ID: 3d28f338, Job ID: b8a4ca42-f9ab-4568-bf26-41e0043d9524, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:28:04,494 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [3d28f338] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:28:04,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [3d28f338] Correlation ID: 138ac13f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:28:04,505 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-96) [65a380bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:28:04,506 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-96) [5d1ad434] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:28:04,507 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [5d1ad434] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [1ed1d7d] Autorecovering 1 hosts >2014-04-29 10:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [1ed1d7d] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3ae24d9b >2014-04-29 10:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] FINISH, SetVdsStatusVDSCommand, log id: 3ae24d9b >2014-04-29 10:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3a9d9bb3 >2014-04-29 10:30:00,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1d7852e7 >2014-04-29 10:30:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] Failed in GetHardwareInfoVDS method >2014-04-29 10:30:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:30:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@468acace >2014-04-29 10:30:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] HostName = 10.34.27.209 >2014-04-29 10:30:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:30:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [655bfdbf] FINISH, GetHardwareInfoVDSCommand, log id: 1d7852e7 >2014-04-29 10:30:00,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [655bfdbf] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:30:00,738 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [43c65f35] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:30:00,740 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:30:00,741 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3a9d9bb3 >2014-04-29 10:30:00,764 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:30:00,765 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:30:01,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 35d184b9 >2014-04-29 10:30:02,105 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] Failed in GetHardwareInfoVDS method >2014-04-29 10:30:02,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:30:02,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@425adad1 >2014-04-29 10:30:02,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] HostName = 10.34.27.209 >2014-04-29 10:30:02,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:30:02,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [34c20962] FINISH, GetHardwareInfoVDSCommand, log id: 35d184b9 >2014-04-29 10:30:02,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [34c20962] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:30:02,120 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [34c20962] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:30:02,335 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-74) [1dfefe79] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:30:02,337 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-74) [10141575] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:30:02,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-74) [10141575] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 575b8e69 >2014-04-29 10:33:02,347 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-74) [10141575] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:33:02,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-74) [10141575] FINISH, ConnectStorageServerVDSCommand, log id: 575b8e69 >2014-04-29 10:33:02,347 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-74) [10141575] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:33:02,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 8b65992 >2014-04-29 10:33:02,825 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 8b65992 >2014-04-29 10:33:02,825 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:33:02,840 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-74) [1b4c97c5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:33:02,841 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) [1b4c97c5] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5ab3a8dd >2014-04-29 10:33:02,845 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) [1b4c97c5] FINISH, SetVdsStatusVDSCommand, log id: 5ab3a8dd >2014-04-29 10:33:02,851 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [1b4c97c5] Correlation ID: 1b4c97c5, Job ID: b2f15f44-6a75-4fb8-b39f-72566b38d549, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:33:02,857 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [1b4c97c5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:33:02,859 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [1b4c97c5] Correlation ID: 1dfefe79, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:33:02,864 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-74) [3251e112] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:33:02,865 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-74) [64ca0b80] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:33:02,865 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) [64ca0b80] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-28) Autorecovering 1 hosts >2014-04-29 10:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-28) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 19a84c68 >2014-04-29 10:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] FINISH, SetVdsStatusVDSCommand, log id: 19a84c68 >2014-04-29 10:35:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3b6bf2cf >2014-04-29 10:35:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4f7c85c6 >2014-04-29 10:35:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] Failed in GetHardwareInfoVDS method >2014-04-29 10:35:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:35:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7832b2ee >2014-04-29 10:35:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] HostName = 10.34.27.209 >2014-04-29 10:35:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:35:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [16f89923] FINISH, GetHardwareInfoVDSCommand, log id: 4f7c85c6 >2014-04-29 10:35:00,709 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [16f89923] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:35:00,720 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-28) [384ae70d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:35:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-28) [27211418] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:35:00,723 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) [27211418] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3b6bf2cf >2014-04-29 10:35:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [27211418] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:35:00,744 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [27211418] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:35:03,210 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4038fb48 >2014-04-29 10:35:03,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) Failed in GetHardwareInfoVDS method >2014-04-29 10:35:03,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:35:03,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5cf954e8 >2014-04-29 10:35:03,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) HostName = 10.34.27.209 >2014-04-29 10:35:03,428 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:35:03,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) FINISH, GetHardwareInfoVDSCommand, log id: 4038fb48 >2014-04-29 10:35:03,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:35:03,445 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:35:03,664 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [2c403452] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:35:03,666 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [66e08f87] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:35:03,673 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [66e08f87] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3e9b8057 >2014-04-29 10:36:04,981 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] Start refreshing all users data >2014-04-29 10:36:07,212 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [62a73d1c] Attempting to update VMs/Templates Ovf. >2014-04-29 10:36:07,213 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [62a73d1c] Attempting to update VM OVFs in Data Center NFS >2014-04-29 10:36:07,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [62a73d1c] Successfully updated VM OVFs in Data Center NFS >2014-04-29 10:36:07,218 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [62a73d1c] Attempting to update template OVFs in Data Center NFS >2014-04-29 10:36:07,220 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [62a73d1c] Successfully updated templates OVFs in Data Center NFS >2014-04-29 10:36:07,220 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [62a73d1c] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 10:36:07,221 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-7) [62a73d1c] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 10:38:03,674 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [66e08f87] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:38:03,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [66e08f87] FINISH, ConnectStorageServerVDSCommand, log id: 3e9b8057 >2014-04-29 10:38:03,674 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [66e08f87] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:38:03,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: aaec344 >2014-04-29 10:38:04,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: aaec344 >2014-04-29 10:38:04,110 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:38:04,123 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-86) [7d05a757] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:38:04,124 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [7d05a757] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2f0584f >2014-04-29 10:38:04,127 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [7d05a757] FINISH, SetVdsStatusVDSCommand, log id: 2f0584f >2014-04-29 10:38:04,133 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [7d05a757] Correlation ID: 7d05a757, Job ID: 102b15ae-7931-43fb-a274-4864ae2d4cb0, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:38:04,140 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [7d05a757] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:38:04,142 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [7d05a757] Correlation ID: 2c403452, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:38:04,146 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [21ab327a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:38:04,147 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [7027322b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:38:04,148 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [7027322b] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-44) [2619bd68] Autorecovering 1 hosts >2014-04-29 10:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-44) [2619bd68] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [61473682] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [61473682] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [61473682] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [61473682] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [61473682] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 271748a6 >2014-04-29 10:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [61473682] FINISH, SetVdsStatusVDSCommand, log id: 271748a6 >2014-04-29 10:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) [61473682] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 78052e9d >2014-04-29 10:40:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [61473682] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2c7a11c1 >2014-04-29 10:40:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [61473682] Failed in GetHardwareInfoVDS method >2014-04-29 10:40:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [61473682] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:40:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [61473682] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@223acb7 >2014-04-29 10:40:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [61473682] HostName = 10.34.27.209 >2014-04-29 10:40:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [61473682] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:40:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [61473682] FINISH, GetHardwareInfoVDSCommand, log id: 2c7a11c1 >2014-04-29 10:40:00,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [61473682] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:40:00,705 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [2d9f5028] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:40:00,706 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [4519be7f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:40:00,707 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) [4519be7f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 78052e9d >2014-04-29 10:40:00,720 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [4519be7f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:40:00,722 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [4519be7f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:40:01,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [1b9561fe] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6f9369a0 >2014-04-29 10:40:01,735 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [1b9561fe] Failed in GetHardwareInfoVDS method >2014-04-29 10:40:01,735 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [1b9561fe] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:40:01,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [1b9561fe] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@11287625 >2014-04-29 10:40:01,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [1b9561fe] HostName = 10.34.27.209 >2014-04-29 10:40:01,735 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [1b9561fe] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:40:01,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [1b9561fe] FINISH, GetHardwareInfoVDSCommand, log id: 6f9369a0 >2014-04-29 10:40:01,740 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [1b9561fe] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:40:01,750 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [1b9561fe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:40:01,968 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-79) [7a8acb94] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:40:01,969 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-79) [59afb097] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:40:01,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [59afb097] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5e2abbb5 >2014-04-29 10:43:01,977 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [59afb097] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:43:01,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [59afb097] FINISH, ConnectStorageServerVDSCommand, log id: 5e2abbb5 >2014-04-29 10:43:01,978 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-79) [59afb097] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:43:01,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 72e6efc9 >2014-04-29 10:43:02,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 72e6efc9 >2014-04-29 10:43:02,413 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:43:02,428 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-79) [74f0ed1b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:43:02,429 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [74f0ed1b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5a291670 >2014-04-29 10:43:02,431 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [74f0ed1b] FINISH, SetVdsStatusVDSCommand, log id: 5a291670 >2014-04-29 10:43:02,438 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [74f0ed1b] Correlation ID: 74f0ed1b, Job ID: 6c1f6b87-8ee6-4f49-a77d-51e8631496c4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:43:02,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [74f0ed1b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:43:02,450 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [74f0ed1b] Correlation ID: 7a8acb94, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:43:02,454 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [43e8ed58] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:43:02,455 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:43:02,456 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-40) Autorecovering 1 hosts >2014-04-29 10:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-40) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3a781c82 >2014-04-29 10:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] FINISH, SetVdsStatusVDSCommand, log id: 3a781c82 >2014-04-29 10:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5e813176 >2014-04-29 10:45:00,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4aa20abe >2014-04-29 10:45:00,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] Failed in GetHardwareInfoVDS method >2014-04-29 10:45:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:45:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4fa8f00 >2014-04-29 10:45:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] HostName = 10.34.27.209 >2014-04-29 10:45:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:45:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [357374ea] FINISH, GetHardwareInfoVDSCommand, log id: 4aa20abe >2014-04-29 10:45:00,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [357374ea] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:45:00,701 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-40) [112bb055] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:45:00,702 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:45:00,703 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5e813176 >2014-04-29 10:45:00,717 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:45:00,718 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:45:02,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1ed28734 >2014-04-29 10:45:03,034 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) Failed in GetHardwareInfoVDS method >2014-04-29 10:45:03,035 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:45:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@758eb31c >2014-04-29 10:45:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) HostName = 10.34.27.209 >2014-04-29 10:45:03,035 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:45:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) FINISH, GetHardwareInfoVDSCommand, log id: 1ed28734 >2014-04-29 10:45:03,039 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:45:03,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:45:03,265 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-26) [1b758b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:45:03,266 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [601691fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:45:03,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [601691fb] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4cadf05c >2014-04-29 10:48:03,274 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [601691fb] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:48:03,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [601691fb] FINISH, ConnectStorageServerVDSCommand, log id: 4cadf05c >2014-04-29 10:48:03,274 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [601691fb] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:48:03,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4e4a9408 >2014-04-29 10:48:03,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 4e4a9408 >2014-04-29 10:48:03,709 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:48:03,720 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-26) [74bd3149] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:48:03,721 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [74bd3149] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: ac27682 >2014-04-29 10:48:03,723 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [74bd3149] FINISH, SetVdsStatusVDSCommand, log id: ac27682 >2014-04-29 10:48:03,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [74bd3149] Correlation ID: 74bd3149, Job ID: 219238dd-ba50-4853-aab6-9d926b11c70e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:48:03,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [74bd3149] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:48:03,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [74bd3149] Correlation ID: 1b758b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:48:03,745 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-26) [6fa999d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:48:03,746 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-26) [1584fc64] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:48:03,747 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [1584fc64] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-80) [365e941f] Autorecovering 1 hosts >2014-04-29 10:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-80) [365e941f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:50:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:50:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:50:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:50:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5c1e14c >2014-04-29 10:50:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] FINISH, SetVdsStatusVDSCommand, log id: 5c1e14c >2014-04-29 10:50:00,018 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 70e793ea >2014-04-29 10:50:00,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 34bb57f6 >2014-04-29 10:50:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] Failed in GetHardwareInfoVDS method >2014-04-29 10:50:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:50:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2cd0ab04 >2014-04-29 10:50:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] HostName = 10.34.27.209 >2014-04-29 10:50:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:50:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [12799b4] FINISH, GetHardwareInfoVDSCommand, log id: 34bb57f6 >2014-04-29 10:50:00,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [12799b4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:50:00,714 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-80) [3b0119de] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:50:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:50:00,716 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 70e793ea >2014-04-29 10:50:00,732 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:50:00,735 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:50:01,147 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1d055733 >2014-04-29 10:50:01,370 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] Failed in GetHardwareInfoVDS method >2014-04-29 10:50:01,370 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:50:01,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@499fe427 >2014-04-29 10:50:01,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] HostName = 10.34.27.209 >2014-04-29 10:50:01,370 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:50:01,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [3425a5d0] FINISH, GetHardwareInfoVDSCommand, log id: 1d055733 >2014-04-29 10:50:01,375 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [3425a5d0] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:50:01,387 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [3425a5d0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:50:01,615 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-2) [24d12373] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:50:01,621 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-2) [771a19f2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:50:01,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-2) [771a19f2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 479c775b >2014-04-29 10:53:01,629 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-2) [771a19f2] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:53:01,629 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-2) [771a19f2] FINISH, ConnectStorageServerVDSCommand, log id: 479c775b >2014-04-29 10:53:01,629 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-2) [771a19f2] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:53:01,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 278f9e3b >2014-04-29 10:53:02,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 278f9e3b >2014-04-29 10:53:02,098 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:53:02,111 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-2) [8d6c239] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:53:02,113 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [8d6c239] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6745f9d0 >2014-04-29 10:53:02,115 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [8d6c239] FINISH, SetVdsStatusVDSCommand, log id: 6745f9d0 >2014-04-29 10:53:02,123 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [8d6c239] Correlation ID: 8d6c239, Job ID: 54b619e0-f090-4da3-8874-3168f605f604, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:53:02,131 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [8d6c239] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:53:02,133 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [8d6c239] Correlation ID: 24d12373, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:53:02,141 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-2) [6ef1804f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:53:02,142 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-2) [56601b24] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:53:02,143 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [56601b24] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 10:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-60) Autorecovering 1 hosts >2014-04-29 10:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-60) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 10:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 10:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 10:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7f24a77a >2014-04-29 10:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] FINISH, SetVdsStatusVDSCommand, log id: 7f24a77a >2014-04-29 10:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 47a346a2 >2014-04-29 10:55:00,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2a4746b1 >2014-04-29 10:55:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] Failed in GetHardwareInfoVDS method >2014-04-29 10:55:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:55:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2d722426 >2014-04-29 10:55:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] HostName = 10.34.27.209 >2014-04-29 10:55:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:55:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [3461113a] FINISH, GetHardwareInfoVDSCommand, log id: 2a4746b1 >2014-04-29 10:55:00,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [3461113a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:55:00,710 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-60) [7f0e084d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:55:00,711 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-60) [b680a50] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:55:00,712 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) [b680a50] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 47a346a2 >2014-04-29 10:55:00,729 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [b680a50] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 10:55:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [b680a50] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 10:55:02,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3a9c757b >2014-04-29 10:55:02,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) Failed in GetHardwareInfoVDS method >2014-04-29 10:55:02,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:55:02,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@75e4fc4c >2014-04-29 10:55:02,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) HostName = 10.34.27.209 >2014-04-29 10:55:02,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 10:55:02,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) FINISH, GetHardwareInfoVDSCommand, log id: 3a9c757b >2014-04-29 10:55:02,709 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 10:55:02,723 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 10:55:02,939 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-37) [4b40806a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:55:02,941 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-37) [47436ff4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 10:55:02,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [47436ff4] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 63f47f45 >2014-04-29 10:58:02,948 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [47436ff4] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 10:58:02,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [47436ff4] FINISH, ConnectStorageServerVDSCommand, log id: 63f47f45 >2014-04-29 10:58:02,948 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-37) [47436ff4] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 10:58:02,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 602dd3e0 >2014-04-29 10:58:03,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 602dd3e0 >2014-04-29 10:58:03,411 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 10:58:03,423 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-37) [5a418131] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:58:03,425 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [5a418131] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 12452ca2 >2014-04-29 10:58:03,428 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [5a418131] FINISH, SetVdsStatusVDSCommand, log id: 12452ca2 >2014-04-29 10:58:03,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [5a418131] Correlation ID: 5a418131, Job ID: ea84f798-c174-49ae-8c34-1c613a877e23, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 10:58:03,440 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [5a418131] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 10:58:03,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [5a418131] Correlation ID: 4b40806a, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 10:58:03,449 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-37) [f8a83ec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:58:03,450 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 10:58:03,451 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) [38b1be3b] Autorecovering 1 hosts >2014-04-29 11:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) [38b1be3b] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 39648ab3 >2014-04-29 11:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] FINISH, SetVdsStatusVDSCommand, log id: 39648ab3 >2014-04-29 11:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1ab98789 >2014-04-29 11:00:00,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6fa78ef9 >2014-04-29 11:00:00,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] Failed in GetHardwareInfoVDS method >2014-04-29 11:00:00,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:00:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@619bb6b3 >2014-04-29 11:00:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] HostName = 10.34.27.209 >2014-04-29 11:00:00,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:00:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] FINISH, GetHardwareInfoVDSCommand, log id: 6fa78ef9 >2014-04-29 11:00:00,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [7e0a6fb1] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:00:00,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-29) [24d73d8f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:00:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:00:00,710 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1ab98789 >2014-04-29 11:00:00,725 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:00:00,726 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:00:03,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [64c6b82] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3f34a585 >2014-04-29 11:00:04,044 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [64c6b82] Failed in GetHardwareInfoVDS method >2014-04-29 11:00:04,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [64c6b82] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:00:04,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [64c6b82] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@195edf2e >2014-04-29 11:00:04,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [64c6b82] HostName = 10.34.27.209 >2014-04-29 11:00:04,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [64c6b82] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:00:04,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [64c6b82] FINISH, GetHardwareInfoVDSCommand, log id: 3f34a585 >2014-04-29 11:00:04,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [64c6b82] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:00:04,059 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [64c6b82] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:00:04,280 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-34) [29bb102a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:00:04,283 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [47958412] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:00:04,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [47958412] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1a79bc2c >2014-04-29 11:03:04,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [47958412] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:03:04,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [47958412] FINISH, ConnectStorageServerVDSCommand, log id: 1a79bc2c >2014-04-29 11:03:04,296 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [47958412] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:03:04,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 38b3ba6f >2014-04-29 11:03:04,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 38b3ba6f >2014-04-29 11:03:04,741 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:03:04,753 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-34) [4ab66e39] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:03:04,754 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [4ab66e39] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 13cf3b5e >2014-04-29 11:03:04,759 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [4ab66e39] FINISH, SetVdsStatusVDSCommand, log id: 13cf3b5e >2014-04-29 11:03:04,764 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [4ab66e39] Correlation ID: 4ab66e39, Job ID: b190724a-f491-4bda-9836-d60dcbfc24a8, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:03:04,769 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [4ab66e39] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:03:04,770 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [4ab66e39] Correlation ID: 29bb102a, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:03:04,774 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [2042543d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:03:04,775 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [56c15d09] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:03:04,776 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [56c15d09] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-75) [76668959] Autorecovering 1 hosts >2014-04-29 11:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-75) [76668959] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [76655763] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [76655763] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [76655763] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [76655763] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [76655763] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1a7a5f32 >2014-04-29 11:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [76655763] FINISH, SetVdsStatusVDSCommand, log id: 1a7a5f32 >2014-04-29 11:05:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) [76655763] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 33131c38 >2014-04-29 11:05:00,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [76655763] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 64805639 >2014-04-29 11:05:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [76655763] Failed in GetHardwareInfoVDS method >2014-04-29 11:05:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [76655763] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:05:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [76655763] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@259cebbb >2014-04-29 11:05:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [76655763] HostName = 10.34.27.209 >2014-04-29 11:05:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [76655763] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:05:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [76655763] FINISH, GetHardwareInfoVDSCommand, log id: 64805639 >2014-04-29 11:05:00,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [76655763] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:05:00,766 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-75) [19963d15] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:05:00,768 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:05:00,769 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 33131c38 >2014-04-29 11:05:00,785 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:05:00,786 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:05:02,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 714028fa >2014-04-29 11:05:02,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) Failed in GetHardwareInfoVDS method >2014-04-29 11:05:02,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:05:02,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2b426f2c >2014-04-29 11:05:02,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) HostName = 10.34.27.209 >2014-04-29 11:05:02,382 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:05:02,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) FINISH, GetHardwareInfoVDSCommand, log id: 714028fa >2014-04-29 11:05:02,387 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:05:02,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:05:02,627 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [48ad5d17] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:05:02,628 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [5ac795a5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:05:02,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [5ac795a5] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 51ec7685 >2014-04-29 11:08:02,638 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [5ac795a5] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:08:02,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [5ac795a5] FINISH, ConnectStorageServerVDSCommand, log id: 51ec7685 >2014-04-29 11:08:02,639 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [5ac795a5] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:08:02,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 28a3eb94 >2014-04-29 11:08:03,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 28a3eb94 >2014-04-29 11:08:03,273 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:08:03,285 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-20) [4ac84fac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:08:03,286 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [4ac84fac] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 679e9c8 >2014-04-29 11:08:03,289 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [4ac84fac] FINISH, SetVdsStatusVDSCommand, log id: 679e9c8 >2014-04-29 11:08:03,298 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [4ac84fac] Correlation ID: 4ac84fac, Job ID: 06412071-462f-4f3d-ab43-658210d0879e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:08:03,305 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [4ac84fac] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:08:03,310 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [4ac84fac] Correlation ID: 48ad5d17, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:08:03,320 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [32d39361] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:08:03,321 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [6837cb5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:08:03,322 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) [6837cb5] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) Autorecovering 1 hosts >2014-04-29 11:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 67cde0b7 >2014-04-29 11:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] FINISH, SetVdsStatusVDSCommand, log id: 67cde0b7 >2014-04-29 11:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 57b1dd3a >2014-04-29 11:10:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1cc69dba >2014-04-29 11:10:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] Failed in GetHardwareInfoVDS method >2014-04-29 11:10:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:10:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3f8abf88 >2014-04-29 11:10:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] HostName = 10.34.27.209 >2014-04-29 11:10:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:10:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] FINISH, GetHardwareInfoVDSCommand, log id: 1cc69dba >2014-04-29 11:10:00,710 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [3b6b2ec6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:10:00,720 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [7b93c2bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:10:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:10:00,725 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 57b1dd3a >2014-04-29 11:10:00,748 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:10:00,749 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:10:03,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6e218b56 >2014-04-29 11:10:03,890 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) Failed in GetHardwareInfoVDS method >2014-04-29 11:10:03,890 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:10:03,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5c42f396 >2014-04-29 11:10:03,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) HostName = 10.34.27.209 >2014-04-29 11:10:03,890 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:10:03,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) FINISH, GetHardwareInfoVDSCommand, log id: 6e218b56 >2014-04-29 11:10:03,898 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:10:03,906 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:10:04,130 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [6e612981] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:10:04,132 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [61b6ebc3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:10:04,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [61b6ebc3] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5bd309f6 >2014-04-29 11:13:04,140 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [61b6ebc3] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:13:04,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [61b6ebc3] FINISH, ConnectStorageServerVDSCommand, log id: 5bd309f6 >2014-04-29 11:13:04,140 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [61b6ebc3] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:13:04,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 733bd124 >2014-04-29 11:13:04,576 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 733bd124 >2014-04-29 11:13:04,576 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:13:04,590 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-23) [70ec1835] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:13:04,591 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [70ec1835] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 161b4a69 >2014-04-29 11:13:04,593 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [70ec1835] FINISH, SetVdsStatusVDSCommand, log id: 161b4a69 >2014-04-29 11:13:04,600 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [70ec1835] Correlation ID: 70ec1835, Job ID: 6fbb380b-5bfc-4758-9529-5dc5a3125935, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:13:04,605 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [70ec1835] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:13:04,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [70ec1835] Correlation ID: 6e612981, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:13:04,610 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-23) [2d385c5b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:13:04,611 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-23) [ea7eac6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:13:04,612 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) [ea7eac6] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) [1909a3ee] Autorecovering 1 hosts >2014-04-29 11:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) [1909a3ee] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2290458c >2014-04-29 11:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] FINISH, SetVdsStatusVDSCommand, log id: 2290458c >2014-04-29 11:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 687985c6 >2014-04-29 11:15:00,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7abf0c80 >2014-04-29 11:15:00,677 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] Failed in GetHardwareInfoVDS method >2014-04-29 11:15:00,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:15:00,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@561ecbe1 >2014-04-29 11:15:00,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] HostName = 10.34.27.209 >2014-04-29 11:15:00,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:15:00,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5f16599c] FINISH, GetHardwareInfoVDSCommand, log id: 7abf0c80 >2014-04-29 11:15:00,682 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [5f16599c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:15:00,690 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [6b864b3f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:15:00,691 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:15:00,692 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 687985c6 >2014-04-29 11:15:00,705 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:15:00,705 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:15:01,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [8d93127] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 750458fe >2014-04-29 11:15:02,193 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [8d93127] Failed in GetHardwareInfoVDS method >2014-04-29 11:15:02,193 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [8d93127] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:15:02,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [8d93127] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@706ec7b6 >2014-04-29 11:15:02,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [8d93127] HostName = 10.34.27.209 >2014-04-29 11:15:02,194 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [8d93127] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:15:02,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [8d93127] FINISH, GetHardwareInfoVDSCommand, log id: 750458fe >2014-04-29 11:15:02,198 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [8d93127] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:15:02,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [8d93127] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:15:02,430 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-19) [446df5a6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:15:02,432 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [6e7f783c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:15:02,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [6e7f783c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: bf032dd >2014-04-29 11:18:02,440 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [6e7f783c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:18:02,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [6e7f783c] FINISH, ConnectStorageServerVDSCommand, log id: bf032dd >2014-04-29 11:18:02,440 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [6e7f783c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:18:02,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 149ec832 >2014-04-29 11:18:02,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 149ec832 >2014-04-29 11:18:02,930 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:18:02,942 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-19) [3c3e4d67] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:18:02,943 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [3c3e4d67] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3cf6ceb4 >2014-04-29 11:18:02,946 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [3c3e4d67] FINISH, SetVdsStatusVDSCommand, log id: 3cf6ceb4 >2014-04-29 11:18:02,953 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [3c3e4d67] Correlation ID: 3c3e4d67, Job ID: 08eb8489-813b-4821-9da5-85092f7a3702, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:18:02,960 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [3c3e4d67] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:18:02,962 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [3c3e4d67] Correlation ID: 446df5a6, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:18:02,965 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-19) [71c1cb30] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:18:02,967 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:18:02,967 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-92) [5e7dac47] Autorecovering 1 hosts >2014-04-29 11:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-92) [5e7dac47] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:20:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:20:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1ef64df6 >2014-04-29 11:20:00,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] FINISH, SetVdsStatusVDSCommand, log id: 1ef64df6 >2014-04-29 11:20:00,018 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 11d5f417 >2014-04-29 11:20:00,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3101d289 >2014-04-29 11:20:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] Failed in GetHardwareInfoVDS method >2014-04-29 11:20:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:20:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@21627ce7 >2014-04-29 11:20:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] HostName = 10.34.27.209 >2014-04-29 11:20:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:20:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [724369ae] FINISH, GetHardwareInfoVDSCommand, log id: 3101d289 >2014-04-29 11:20:00,694 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [724369ae] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:20:00,703 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-92) [5675d45b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:20:00,704 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-92) [1f47c9fd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:20:00,705 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) [1f47c9fd] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 11d5f417 >2014-04-29 11:20:00,722 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [1f47c9fd] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:20:00,723 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [1f47c9fd] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:20:03,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 12736e64 >2014-04-29 11:20:03,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) Failed in GetHardwareInfoVDS method >2014-04-29 11:20:03,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:20:03,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@585470b2 >2014-04-29 11:20:03,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) HostName = 10.34.27.209 >2014-04-29 11:20:03,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:20:03,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) FINISH, GetHardwareInfoVDSCommand, log id: 12736e64 >2014-04-29 11:20:03,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:20:03,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:20:03,783 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-22) [3d38d3ef] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:20:03,785 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [6e0db621] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:20:03,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e0db621] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3bfa1084 >2014-04-29 11:23:03,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e0db621] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:23:03,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e0db621] FINISH, ConnectStorageServerVDSCommand, log id: 3bfa1084 >2014-04-29 11:23:03,794 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [6e0db621] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:23:03,797 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5569e5ca >2014-04-29 11:23:04,229 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 5569e5ca >2014-04-29 11:23:04,229 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:23:04,242 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-22) [6b30c35] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:23:04,243 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [6b30c35] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 249935de >2014-04-29 11:23:04,245 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [6b30c35] FINISH, SetVdsStatusVDSCommand, log id: 249935de >2014-04-29 11:23:04,251 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [6b30c35] Correlation ID: 6b30c35, Job ID: 623075d2-05b2-448c-9a09-6225ff84fd6a, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:23:04,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [6b30c35] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:23:04,258 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [6b30c35] Correlation ID: 3d38d3ef, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:23:04,263 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-22) [7a1663c9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:23:04,264 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-22) [63759f74] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:23:04,265 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [63759f74] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Autorecovering 1 hosts >2014-04-29 11:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-3) [7d0c4c23] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 67b547c2 >2014-04-29 11:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] FINISH, SetVdsStatusVDSCommand, log id: 67b547c2 >2014-04-29 11:25:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: ddd09d1 >2014-04-29 11:25:00,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5758fc5e >2014-04-29 11:25:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] Failed in GetHardwareInfoVDS method >2014-04-29 11:25:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:25:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1320ac1f >2014-04-29 11:25:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] HostName = 10.34.27.209 >2014-04-29 11:25:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:25:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [56c69575] FINISH, GetHardwareInfoVDSCommand, log id: 5758fc5e >2014-04-29 11:25:00,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [56c69575] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:25:00,720 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-3) [3acc3c86] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:25:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:25:00,723 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) [7a0debcc] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: ddd09d1 >2014-04-29 11:25:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:25:00,743 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:25:01,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6837cb5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 676abc02 >2014-04-29 11:25:01,892 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6837cb5] Failed in GetHardwareInfoVDS method >2014-04-29 11:25:01,892 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6837cb5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:25:01,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6837cb5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@25304dc0 >2014-04-29 11:25:01,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6837cb5] HostName = 10.34.27.209 >2014-04-29 11:25:01,892 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6837cb5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:25:01,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6837cb5] FINISH, GetHardwareInfoVDSCommand, log id: 676abc02 >2014-04-29 11:25:01,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [6837cb5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:25:01,904 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [6837cb5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:25:02,156 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [3ea2a39d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:25:02,157 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [72fe7a63] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:25:02,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [72fe7a63] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 685c4b79 >2014-04-29 11:28:02,165 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [72fe7a63] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:28:02,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [72fe7a63] FINISH, ConnectStorageServerVDSCommand, log id: 685c4b79 >2014-04-29 11:28:02,165 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [72fe7a63] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:28:02,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 763029eb >2014-04-29 11:28:02,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 763029eb >2014-04-29 11:28:02,609 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:28:02,622 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-20) [3d39e2e3] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:28:02,623 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [3d39e2e3] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 144b5454 >2014-04-29 11:28:02,626 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [3d39e2e3] FINISH, SetVdsStatusVDSCommand, log id: 144b5454 >2014-04-29 11:28:02,633 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [3d39e2e3] Correlation ID: 3d39e2e3, Job ID: a78898ad-54dc-4aa4-9e6f-9f66a83ddbf3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:28:02,640 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [3d39e2e3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:28:02,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [3d39e2e3] Correlation ID: 3ea2a39d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:28:02,658 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [3fe9a2bb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:28:02,659 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [797cbadd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:28:02,660 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) [797cbadd] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) Autorecovering 1 hosts >2014-04-29 11:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 36fdccab >2014-04-29 11:30:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] FINISH, SetVdsStatusVDSCommand, log id: 36fdccab >2014-04-29 11:30:00,019 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4f7ba6bd >2014-04-29 11:30:00,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3c24960e >2014-04-29 11:30:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] Failed in GetHardwareInfoVDS method >2014-04-29 11:30:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:30:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5f78ffb >2014-04-29 11:30:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] HostName = 10.34.27.209 >2014-04-29 11:30:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:30:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4cc7e346] FINISH, GetHardwareInfoVDSCommand, log id: 3c24960e >2014-04-29 11:30:00,697 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4cc7e346] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:30:00,707 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [3b1865a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:30:00,708 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:30:00,709 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4f7ba6bd >2014-04-29 11:30:00,723 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:30:00,723 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:30:03,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [cff75eb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6afeb7aa >2014-04-29 11:30:03,227 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [cff75eb] Failed in GetHardwareInfoVDS method >2014-04-29 11:30:03,227 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [cff75eb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:30:03,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [cff75eb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@78cc30c6 >2014-04-29 11:30:03,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [cff75eb] HostName = 10.34.27.209 >2014-04-29 11:30:03,227 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [cff75eb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:30:03,227 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [cff75eb] FINISH, GetHardwareInfoVDSCommand, log id: 6afeb7aa >2014-04-29 11:30:03,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [cff75eb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:30:03,242 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [cff75eb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:30:03,473 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-11) [1614e9dc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:30:03,478 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-11) [75b34a0b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:30:03,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [75b34a0b] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b7fe801 >2014-04-29 11:33:03,487 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [75b34a0b] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:33:03,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [75b34a0b] FINISH, ConnectStorageServerVDSCommand, log id: 7b7fe801 >2014-04-29 11:33:03,488 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-11) [75b34a0b] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:33:03,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 77e13e1a >2014-04-29 11:33:03,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 77e13e1a >2014-04-29 11:33:03,935 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:33:03,945 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-11) [3d3b084e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:33:03,946 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [3d3b084e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2404faa5 >2014-04-29 11:33:03,948 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [3d3b084e] FINISH, SetVdsStatusVDSCommand, log id: 2404faa5 >2014-04-29 11:33:03,954 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [3d3b084e] Correlation ID: 3d3b084e, Job ID: 89df2d1c-1e9f-48a2-96b7-6d4837b67cbe, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:33:03,965 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [3d3b084e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:33:03,967 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [3d3b084e] Correlation ID: 1614e9dc, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:33:03,972 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-11) [13cfb95] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:33:03,973 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:33:03,974 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-100) Autorecovering 1 hosts >2014-04-29 11:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-100) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 75788c02 >2014-04-29 11:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] FINISH, SetVdsStatusVDSCommand, log id: 75788c02 >2014-04-29 11:35:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4486bd2a >2014-04-29 11:35:00,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1634ee34 >2014-04-29 11:35:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] Failed in GetHardwareInfoVDS method >2014-04-29 11:35:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:35:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6d02a34f >2014-04-29 11:35:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] HostName = 10.34.27.209 >2014-04-29 11:35:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:35:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [116ce853] FINISH, GetHardwareInfoVDSCommand, log id: 1634ee34 >2014-04-29 11:35:00,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [116ce853] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:35:00,745 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-100) [7d8e7042] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:35:00,746 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:35:00,747 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4486bd2a >2014-04-29 11:35:00,769 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:35:00,770 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:35:01,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 51d8a430 >2014-04-29 11:35:01,543 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) Failed in GetHardwareInfoVDS method >2014-04-29 11:35:01,543 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:35:01,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6603754a >2014-04-29 11:35:01,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) HostName = 10.34.27.209 >2014-04-29 11:35:01,543 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:35:01,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) FINISH, GetHardwareInfoVDSCommand, log id: 51d8a430 >2014-04-29 11:35:01,548 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:35:01,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:35:01,773 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-36) [4ea729ec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:35:01,774 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [739b8764] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:35:01,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [739b8764] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5804041b >2014-04-29 11:36:04,984 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-17) Start refreshing all users data >2014-04-29 11:36:07,222 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Attempting to update VMs/Templates Ovf. >2014-04-29 11:36:07,223 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Attempting to update VM OVFs in Data Center NFS >2014-04-29 11:36:07,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Successfully updated VM OVFs in Data Center NFS >2014-04-29 11:36:07,230 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Attempting to update template OVFs in Data Center NFS >2014-04-29 11:36:07,231 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Successfully updated templates OVFs in Data Center NFS >2014-04-29 11:36:07,231 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 11:36:07,232 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 11:38:01,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [739b8764] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:38:01,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [739b8764] FINISH, ConnectStorageServerVDSCommand, log id: 5804041b >2014-04-29 11:38:01,784 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [739b8764] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:38:01,789 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7feb18ac >2014-04-29 11:38:02,217 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7feb18ac >2014-04-29 11:38:02,217 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:38:02,229 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-36) [260191ff] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:38:02,230 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [260191ff] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1700c7e5 >2014-04-29 11:38:02,232 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [260191ff] FINISH, SetVdsStatusVDSCommand, log id: 1700c7e5 >2014-04-29 11:38:02,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [260191ff] Correlation ID: 260191ff, Job ID: 10b609a6-6074-49c4-a34a-5890ad8490a4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:38:02,244 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [260191ff] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:38:02,246 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [260191ff] Correlation ID: 4ea729ec, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:38:02,249 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [3d9a5aee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:38:02,250 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [2a1d265f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:38:02,251 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) [2a1d265f] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-60) [b680a50] Autorecovering 1 hosts >2014-04-29 11:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-60) [b680a50] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 10dfa179 >2014-04-29 11:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] FINISH, SetVdsStatusVDSCommand, log id: 10dfa179 >2014-04-29 11:40:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5212fc9b >2014-04-29 11:40:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 10dc5d7 >2014-04-29 11:40:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] Failed in GetHardwareInfoVDS method >2014-04-29 11:40:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:40:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2f37a6cb >2014-04-29 11:40:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] HostName = 10.34.27.209 >2014-04-29 11:40:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:40:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] FINISH, GetHardwareInfoVDSCommand, log id: 10dc5d7 >2014-04-29 11:40:00,708 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [36e7e5aa] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:40:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-60) [4f9d45bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:40:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-60) [29a144e6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:40:00,718 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) [29a144e6] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5212fc9b >2014-04-29 11:40:00,736 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [29a144e6] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:40:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [29a144e6] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:40:02,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 20113264 >2014-04-29 11:40:02,818 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] Failed in GetHardwareInfoVDS method >2014-04-29 11:40:02,818 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:40:02,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@796ae9f4 >2014-04-29 11:40:02,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] HostName = 10.34.27.209 >2014-04-29 11:40:02,818 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:40:02,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] FINISH, GetHardwareInfoVDSCommand, log id: 20113264 >2014-04-29 11:40:02,823 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:40:02,832 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [1a6b3c8d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:40:03,050 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-48) [13c95338] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:40:03,051 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [4f0d9318] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:40:03,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [4f0d9318] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2aa58905 >2014-04-29 11:43:03,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [4f0d9318] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:43:03,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [4f0d9318] FINISH, ConnectStorageServerVDSCommand, log id: 2aa58905 >2014-04-29 11:43:03,058 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [4f0d9318] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:43:03,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4fbd8065 >2014-04-29 11:43:03,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 4fbd8065 >2014-04-29 11:43:03,491 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:43:03,503 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-48) [48d5b3a0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:43:03,504 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [48d5b3a0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1a4e0ee3 >2014-04-29 11:43:03,507 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [48d5b3a0] FINISH, SetVdsStatusVDSCommand, log id: 1a4e0ee3 >2014-04-29 11:43:03,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [48d5b3a0] Correlation ID: 48d5b3a0, Job ID: 8c56165c-0dd1-4075-8070-bb7597583e35, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:43:03,520 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [48d5b3a0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:43:03,522 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [48d5b3a0] Correlation ID: 13c95338, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:43:03,527 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [54dc6293] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:43:03,528 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [67a1f47c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:43:03,529 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [67a1f47c] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-34) [56c15d09] Autorecovering 1 hosts >2014-04-29 11:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-34) [56c15d09] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [63693861] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [63693861] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [63693861] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [63693861] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [63693861] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 12bbfdb4 >2014-04-29 11:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [63693861] FINISH, SetVdsStatusVDSCommand, log id: 12bbfdb4 >2014-04-29 11:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) [63693861] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5754e67f >2014-04-29 11:45:00,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [63693861] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 55afd0c3 >2014-04-29 11:45:00,683 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [63693861] Failed in GetHardwareInfoVDS method >2014-04-29 11:45:00,683 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [63693861] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:45:00,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [63693861] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@9b2de28 >2014-04-29 11:45:00,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [63693861] HostName = 10.34.27.209 >2014-04-29 11:45:00,683 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [63693861] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:45:00,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [63693861] FINISH, GetHardwareInfoVDSCommand, log id: 55afd0c3 >2014-04-29 11:45:00,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [63693861] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:45:00,698 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [43e43563] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:45:00,699 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:45:00,700 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5754e67f >2014-04-29 11:45:00,715 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:45:00,716 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:45:03,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 32dfeed >2014-04-29 11:45:04,104 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] Failed in GetHardwareInfoVDS method >2014-04-29 11:45:04,104 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:45:04,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@40bb7b3c >2014-04-29 11:45:04,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] HostName = 10.34.27.209 >2014-04-29 11:45:04,104 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:45:04,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5c90dac9] FINISH, GetHardwareInfoVDSCommand, log id: 32dfeed >2014-04-29 11:45:04,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [5c90dac9] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:45:04,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [5c90dac9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:45:04,341 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-58) [29906902] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:45:04,343 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [471648a2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:45:04,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [471648a2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 53102f31 >2014-04-29 11:48:04,351 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [471648a2] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:48:04,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [471648a2] FINISH, ConnectStorageServerVDSCommand, log id: 53102f31 >2014-04-29 11:48:04,351 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [471648a2] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:48:04,355 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3bed4614 >2014-04-29 11:48:04,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 3bed4614 >2014-04-29 11:48:04,786 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:48:04,797 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-58) [2449c530] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:48:04,798 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [2449c530] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1be407e6 >2014-04-29 11:48:04,800 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [2449c530] FINISH, SetVdsStatusVDSCommand, log id: 1be407e6 >2014-04-29 11:48:04,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [2449c530] Correlation ID: 2449c530, Job ID: 2af08d91-08ff-40d6-ace1-363e103af3c5, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:48:04,816 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [2449c530] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:48:04,817 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [2449c530] Correlation ID: 29906902, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:48:04,823 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [cc6e0c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:48:04,824 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [3d8c60fc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:48:04,825 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [3d8c60fc] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-44) [4519be7f] Autorecovering 1 hosts >2014-04-29 11:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-44) [4519be7f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 58387f70 >2014-04-29 11:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] FINISH, SetVdsStatusVDSCommand, log id: 58387f70 >2014-04-29 11:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7131a684 >2014-04-29 11:50:00,465 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6f1a3b98 >2014-04-29 11:50:00,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] Failed in GetHardwareInfoVDS method >2014-04-29 11:50:00,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:50:00,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@233eb124 >2014-04-29 11:50:00,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] HostName = 10.34.27.209 >2014-04-29 11:50:00,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:50:00,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [201c5160] FINISH, GetHardwareInfoVDSCommand, log id: 6f1a3b98 >2014-04-29 11:50:00,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [201c5160] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:50:00,699 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [32930c6c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:50:00,700 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [8434417] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:50:00,701 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) [8434417] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7131a684 >2014-04-29 11:50:00,717 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [8434417] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:50:00,718 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [8434417] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:50:02,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6d9c6c >2014-04-29 11:50:02,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] Failed in GetHardwareInfoVDS method >2014-04-29 11:50:02,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:50:02,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1c3876ac >2014-04-29 11:50:02,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] HostName = 10.34.27.209 >2014-04-29 11:50:02,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:50:02,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [1f337138] FINISH, GetHardwareInfoVDSCommand, log id: 6d9c6c >2014-04-29 11:50:02,403 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [1f337138] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:50:02,413 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [1f337138] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:50:02,630 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-29) [685d8963] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:50:02,632 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [76f8c017] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:50:02,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [76f8c017] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5ac1d836 >2014-04-29 11:53:02,639 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [76f8c017] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:53:02,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [76f8c017] FINISH, ConnectStorageServerVDSCommand, log id: 5ac1d836 >2014-04-29 11:53:02,639 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [76f8c017] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:53:02,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 47b3f4c6 >2014-04-29 11:53:03,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 47b3f4c6 >2014-04-29 11:53:03,869 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:53:03,880 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-29) [40d9e795] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:53:03,881 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [40d9e795] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3fc5446a >2014-04-29 11:53:03,884 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [40d9e795] FINISH, SetVdsStatusVDSCommand, log id: 3fc5446a >2014-04-29 11:53:03,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [40d9e795] Correlation ID: 40d9e795, Job ID: 07747ce5-3f27-434e-96f6-0452e3fb9de5, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:53:03,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [40d9e795] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:53:03,897 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [40d9e795] Correlation ID: 685d8963, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:53:03,901 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-29) [7c9cb0b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:53:03,902 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-29) [4f947e59] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:53:03,905 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) [4f947e59] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 11:55:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-82) Autorecovering 1 hosts >2014-04-29 11:55:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-82) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 11:55:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:55:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:55:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 11:55:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 11:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2428463e >2014-04-29 11:55:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] FINISH, SetVdsStatusVDSCommand, log id: 2428463e >2014-04-29 11:55:00,018 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 74b9dec4 >2014-04-29 11:55:05,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 49398a02 >2014-04-29 11:55:05,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] Failed in GetHardwareInfoVDS method >2014-04-29 11:55:05,873 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:55:05,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6093aedc >2014-04-29 11:55:05,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] HostName = 10.34.27.209 >2014-04-29 11:55:05,873 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:55:05,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [5acbc40f] FINISH, GetHardwareInfoVDSCommand, log id: 49398a02 >2014-04-29 11:55:05,878 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [5acbc40f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:55:05,886 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-82) [154faf72] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:55:05,887 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-82) [24ac60e8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:55:05,888 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) [24ac60e8] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 74b9dec4 >2014-04-29 11:55:05,902 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [24ac60e8] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 11:55:05,903 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [24ac60e8] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 11:55:07,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5dc89edc >2014-04-29 11:55:07,475 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] Failed in GetHardwareInfoVDS method >2014-04-29 11:55:07,475 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:55:07,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1e35ad9e >2014-04-29 11:55:07,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] HostName = 10.34.27.209 >2014-04-29 11:55:07,475 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 11:55:07,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [35d6890] FINISH, GetHardwareInfoVDSCommand, log id: 5dc89edc >2014-04-29 11:55:07,480 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [35d6890] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 11:55:07,495 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [35d6890] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 11:55:07,711 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-49) [705be8da] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:55:07,712 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-49) [764093d4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 11:55:07,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-49) [764093d4] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 421356ea >2014-04-29 11:58:07,720 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-49) [764093d4] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 11:58:07,720 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-49) [764093d4] FINISH, ConnectStorageServerVDSCommand, log id: 421356ea >2014-04-29 11:58:07,721 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-49) [764093d4] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 11:58:07,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 6b1dad47 >2014-04-29 11:58:08,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 6b1dad47 >2014-04-29 11:58:08,476 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 11:58:08,487 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-49) [5caf4f49] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:58:08,488 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [5caf4f49] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 55d2c8e2 >2014-04-29 11:58:08,492 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [5caf4f49] FINISH, SetVdsStatusVDSCommand, log id: 55d2c8e2 >2014-04-29 11:58:08,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [5caf4f49] Correlation ID: 5caf4f49, Job ID: 8c4b8cf1-8f4f-44c2-bcf8-347b5b8ca5fc, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 11:58:08,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [5caf4f49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 11:58:08,512 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [5caf4f49] Correlation ID: 705be8da, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 11:58:08,516 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-49) [5a2294db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:58:08,517 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-49) [6ed4d8d8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 11:58:08,518 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-49) [6ed4d8d8] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-46) Autorecovering 1 hosts >2014-04-29 12:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-46) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 30415ff5 >2014-04-29 12:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] FINISH, SetVdsStatusVDSCommand, log id: 30415ff5 >2014-04-29 12:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 12384d49 >2014-04-29 12:00:00,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 63940053 >2014-04-29 12:00:00,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] Failed in GetHardwareInfoVDS method >2014-04-29 12:00:00,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:00:00,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@9a17c38 >2014-04-29 12:00:00,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] HostName = 10.34.27.209 >2014-04-29 12:00:00,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:00:00,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] FINISH, GetHardwareInfoVDSCommand, log id: 63940053 >2014-04-29 12:00:00,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [76ee5ed9] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:00:00,701 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-46) [14cfcf51] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:00:00,703 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:00:00,703 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 12384d49 >2014-04-29 12:00:00,716 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:00:00,717 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:00:02,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [27199a6d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 472b01c2 >2014-04-29 12:00:03,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [27199a6d] Failed in GetHardwareInfoVDS method >2014-04-29 12:00:03,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [27199a6d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:00:03,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [27199a6d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@58808c0e >2014-04-29 12:00:03,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [27199a6d] HostName = 10.34.27.209 >2014-04-29 12:00:03,085 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [27199a6d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:00:03,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [27199a6d] FINISH, GetHardwareInfoVDSCommand, log id: 472b01c2 >2014-04-29 12:00:03,089 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [27199a6d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:00:03,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [27199a6d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:00:03,315 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-72) [79e67d42] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:00:03,316 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [37fa6499] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:00:03,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [37fa6499] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5392a3e >2014-04-29 12:03:03,325 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [37fa6499] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:03:03,325 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [37fa6499] FINISH, ConnectStorageServerVDSCommand, log id: 5392a3e >2014-04-29 12:03:03,325 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [37fa6499] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:03:03,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 62eacc37 >2014-04-29 12:03:03,760 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 62eacc37 >2014-04-29 12:03:03,760 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:03:03,772 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-72) [7f1d1d89] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:03:03,773 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [7f1d1d89] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 25df623 >2014-04-29 12:03:03,775 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [7f1d1d89] FINISH, SetVdsStatusVDSCommand, log id: 25df623 >2014-04-29 12:03:03,782 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [7f1d1d89] Correlation ID: 7f1d1d89, Job ID: 6d8d62a8-a19d-4ef1-9bc6-484409544116, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:03:03,787 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [7f1d1d89] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:03:03,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [7f1d1d89] Correlation ID: 79e67d42, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:03:03,795 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-72) [5528ded5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:03:03,796 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-72) [58ce83cd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:03:03,797 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [58ce83cd] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-21) [16d7e943] Autorecovering 1 hosts >2014-04-29 12:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-21) [16d7e943] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5c330cbe >2014-04-29 12:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] FINISH, SetVdsStatusVDSCommand, log id: 5c330cbe >2014-04-29 12:05:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 59736780 >2014-04-29 12:05:00,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4372b84c >2014-04-29 12:05:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] Failed in GetHardwareInfoVDS method >2014-04-29 12:05:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:05:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@20a1a82c >2014-04-29 12:05:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] HostName = 10.34.27.209 >2014-04-29 12:05:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:05:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [13b68d0b] FINISH, GetHardwareInfoVDSCommand, log id: 4372b84c >2014-04-29 12:05:00,700 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [13b68d0b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:05:00,713 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-21) [3ff4b8f5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:05:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:05:00,716 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 59736780 >2014-04-29 12:05:00,734 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:05:00,735 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:05:01,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 453d4177 >2014-04-29 12:05:01,365 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) Failed in GetHardwareInfoVDS method >2014-04-29 12:05:01,365 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:05:01,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@38950f24 >2014-04-29 12:05:01,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) HostName = 10.34.27.209 >2014-04-29 12:05:01,365 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:05:01,366 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) FINISH, GetHardwareInfoVDSCommand, log id: 453d4177 >2014-04-29 12:05:01,371 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:05:01,383 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:05:01,600 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-13) [560903ca] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:05:01,601 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-13) [6ab366fa] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:05:01,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [6ab366fa] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 77756460 >2014-04-29 12:08:01,610 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [6ab366fa] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:08:01,610 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [6ab366fa] FINISH, ConnectStorageServerVDSCommand, log id: 77756460 >2014-04-29 12:08:01,610 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-13) [6ab366fa] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:08:01,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 18987cb9 >2014-04-29 12:08:02,053 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 18987cb9 >2014-04-29 12:08:02,053 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:08:02,064 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-13) [4df8dfed] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:08:02,065 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [4df8dfed] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6c135480 >2014-04-29 12:08:02,069 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [4df8dfed] FINISH, SetVdsStatusVDSCommand, log id: 6c135480 >2014-04-29 12:08:02,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [4df8dfed] Correlation ID: 4df8dfed, Job ID: 826e8e3f-cfd3-498f-89e1-0df14cb8e451, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:08:02,081 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [4df8dfed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:08:02,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [4df8dfed] Correlation ID: 560903ca, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:08:02,087 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-13) [38522943] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:08:02,088 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-13) [b9dcd39] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:08:02,089 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [b9dcd39] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) [4f947e59] Autorecovering 1 hosts >2014-04-29 12:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-29) [4f947e59] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 55385fb2 >2014-04-29 12:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] FINISH, SetVdsStatusVDSCommand, log id: 55385fb2 >2014-04-29 12:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5f232c99 >2014-04-29 12:10:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5899088c >2014-04-29 12:10:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] Failed in GetHardwareInfoVDS method >2014-04-29 12:10:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:10:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@27004cc5 >2014-04-29 12:10:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] HostName = 10.34.27.209 >2014-04-29 12:10:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:10:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6026e0eb] FINISH, GetHardwareInfoVDSCommand, log id: 5899088c >2014-04-29 12:10:00,710 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [6026e0eb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:10:00,719 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-29) [567238a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:10:00,721 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:10:00,721 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5f232c99 >2014-04-29 12:10:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:10:00,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] Correlation ID: 6026e0eb, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was autorecovered. >2014-04-29 12:10:00,739 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:10:02,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 773e02e >2014-04-29 12:10:02,669 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) Failed in GetHardwareInfoVDS method >2014-04-29 12:10:02,669 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:10:02,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4882efd4 >2014-04-29 12:10:02,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) HostName = 10.34.27.209 >2014-04-29 12:10:02,669 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:10:02,669 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) FINISH, GetHardwareInfoVDSCommand, log id: 773e02e >2014-04-29 12:10:02,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:10:02,682 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:10:02,898 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-24) [756812db] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:10:02,900 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [50125cc0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:10:02,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [50125cc0] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 54d09232 >2014-04-29 12:13:02,908 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [50125cc0] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:13:02,908 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [50125cc0] FINISH, ConnectStorageServerVDSCommand, log id: 54d09232 >2014-04-29 12:13:02,908 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [50125cc0] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:13:02,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 403497a8 >2014-04-29 12:13:03,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 403497a8 >2014-04-29 12:13:03,339 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:13:03,350 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-24) [66f766c0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:13:03,351 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [66f766c0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 28688e12 >2014-04-29 12:13:03,354 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [66f766c0] FINISH, SetVdsStatusVDSCommand, log id: 28688e12 >2014-04-29 12:13:03,360 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [66f766c0] Correlation ID: 66f766c0, Job ID: 592ef65d-2b30-4fff-8ffe-3c426b6565da, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:13:03,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [66f766c0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:13:03,367 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [66f766c0] Correlation ID: 756812db, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:13:03,371 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-24) [2cbed003] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:13:03,372 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-24) [1db304d2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:13:03,373 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) [1db304d2] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [688583cd] Autorecovering 1 hosts >2014-04-29 12:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [688583cd] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: b4bdba0 >2014-04-29 12:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] FINISH, SetVdsStatusVDSCommand, log id: b4bdba0 >2014-04-29 12:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 324e8ee1 >2014-04-29 12:15:00,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 558a5eeb >2014-04-29 12:15:00,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] Failed in GetHardwareInfoVDS method >2014-04-29 12:15:00,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:15:00,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6f5dc22c >2014-04-29 12:15:00,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] HostName = 10.34.27.209 >2014-04-29 12:15:00,678 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:15:00,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ad081f8] FINISH, GetHardwareInfoVDSCommand, log id: 558a5eeb >2014-04-29 12:15:00,690 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [7ad081f8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:15:00,702 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-97) [250133ea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:15:00,703 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-97) [6a0c5f21] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:15:00,704 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a0c5f21] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 324e8ee1 >2014-04-29 12:15:00,719 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [6a0c5f21] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:15:00,720 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [6a0c5f21] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:15:03,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 370ee29 >2014-04-29 12:15:03,972 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) Failed in GetHardwareInfoVDS method >2014-04-29 12:15:03,972 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:15:03,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6b23c782 >2014-04-29 12:15:03,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) HostName = 10.34.27.209 >2014-04-29 12:15:03,972 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:15:03,972 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) FINISH, GetHardwareInfoVDSCommand, log id: 370ee29 >2014-04-29 12:15:03,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:15:03,988 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:15:04,217 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-43) [e0f3b47] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:15:04,219 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [22c7b8f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:15:04,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [22c7b8f8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7c58c19e >2014-04-29 12:18:04,228 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [22c7b8f8] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:18:04,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [22c7b8f8] FINISH, ConnectStorageServerVDSCommand, log id: 7c58c19e >2014-04-29 12:18:04,228 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [22c7b8f8] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:18:04,232 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 15567187 >2014-04-29 12:18:04,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 15567187 >2014-04-29 12:18:04,662 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:18:04,675 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-43) [71bd4dda] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:18:04,676 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [71bd4dda] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 14cc21b3 >2014-04-29 12:18:04,680 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [71bd4dda] FINISH, SetVdsStatusVDSCommand, log id: 14cc21b3 >2014-04-29 12:18:04,686 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [71bd4dda] Correlation ID: 71bd4dda, Job ID: b7597853-17b0-4e72-881c-bddf3891370c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:18:04,691 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [71bd4dda] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:18:04,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [71bd4dda] Correlation ID: e0f3b47, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:18:04,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [fa9ba1d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:18:04,710 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [3641f495] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:18:04,711 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [3641f495] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:20:00,034 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-96) [5d1ad434] Autorecovering 1 hosts >2014-04-29 12:20:00,034 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-96) [5d1ad434] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:20:00,036 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:20:00,036 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:20:00,036 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:20:00,036 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:20:00,041 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 671ee440 >2014-04-29 12:20:00,045 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] FINISH, SetVdsStatusVDSCommand, log id: 671ee440 >2014-04-29 12:20:00,047 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 593e827f >2014-04-29 12:20:00,500 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2181685f >2014-04-29 12:20:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] Failed in GetHardwareInfoVDS method >2014-04-29 12:20:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:20:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@36eb4628 >2014-04-29 12:20:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] HostName = 10.34.27.209 >2014-04-29 12:20:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:20:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [564173e5] FINISH, GetHardwareInfoVDSCommand, log id: 2181685f >2014-04-29 12:20:00,725 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [564173e5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:20:00,738 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-96) [308959c4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:20:00,740 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:20:00,741 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 593e827f >2014-04-29 12:20:00,764 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:20:00,765 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:20:02,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: f6394b4 >2014-04-29 12:20:02,276 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] Failed in GetHardwareInfoVDS method >2014-04-29 12:20:02,276 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:20:02,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@52b7bd50 >2014-04-29 12:20:02,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] HostName = 10.34.27.209 >2014-04-29 12:20:02,276 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:20:02,276 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [4bf685f8] FINISH, GetHardwareInfoVDSCommand, log id: f6394b4 >2014-04-29 12:20:02,282 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [4bf685f8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:20:02,291 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [4bf685f8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:20:02,507 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-99) [e5e1f8f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:20:02,510 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-99) [4d63fbd8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:20:02,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-99) [4d63fbd8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5db83ecf >2014-04-29 12:23:02,522 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-99) [4d63fbd8] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:23:02,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-99) [4d63fbd8] FINISH, ConnectStorageServerVDSCommand, log id: 5db83ecf >2014-04-29 12:23:02,522 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-99) [4d63fbd8] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:23:02,526 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 11e306ef >2014-04-29 12:23:02,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 11e306ef >2014-04-29 12:23:02,977 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:23:02,988 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-99) [4c253d7c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:23:02,989 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [4c253d7c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6155717f >2014-04-29 12:23:02,991 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [4c253d7c] FINISH, SetVdsStatusVDSCommand, log id: 6155717f >2014-04-29 12:23:02,996 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [4c253d7c] Correlation ID: 4c253d7c, Job ID: 9b24d04b-26b4-4575-9b02-61dc3b30ceae, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:23:03,002 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [4c253d7c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:23:03,004 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [4c253d7c] Correlation ID: e5e1f8f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:23:03,007 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-99) [24e01cea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:23:03,012 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-99) [147b6be8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:23:03,012 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) [147b6be8] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) [35c89394] Autorecovering 1 hosts >2014-04-29 12:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) [35c89394] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:25:00,024 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 11c943a8 >2014-04-29 12:25:00,044 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] FINISH, SetVdsStatusVDSCommand, log id: 11c943a8 >2014-04-29 12:25:00,045 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7ed7e0e5 >2014-04-29 12:25:00,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 57088861 >2014-04-29 12:25:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] Failed in GetHardwareInfoVDS method >2014-04-29 12:25:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:25:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1ed60db2 >2014-04-29 12:25:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] HostName = 10.34.27.209 >2014-04-29 12:25:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:25:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] FINISH, GetHardwareInfoVDSCommand, log id: 57088861 >2014-04-29 12:25:00,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [2fa2ed5e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:25:00,838 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [76fed4d5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:25:00,840 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [6af5af0a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:25:00,840 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [6af5af0a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7ed7e0e5 >2014-04-29 12:25:00,858 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [6af5af0a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:25:00,859 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [6af5af0a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:25:03,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1dba5b04 >2014-04-29 12:25:03,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) Failed in GetHardwareInfoVDS method >2014-04-29 12:25:03,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:25:03,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@35ad72db >2014-04-29 12:25:03,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) HostName = 10.34.27.209 >2014-04-29 12:25:03,572 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:25:03,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) FINISH, GetHardwareInfoVDSCommand, log id: 1dba5b04 >2014-04-29 12:25:03,577 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:25:03,587 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:25:03,802 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-54) [3fa02f87] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:25:03,804 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-54) [24264165] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:25:03,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-54) [24264165] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 72052267 >2014-04-29 12:28:03,816 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-54) [24264165] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:28:03,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-54) [24264165] FINISH, ConnectStorageServerVDSCommand, log id: 72052267 >2014-04-29 12:28:03,816 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-54) [24264165] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:28:03,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1b0cb574 >2014-04-29 12:28:04,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 1b0cb574 >2014-04-29 12:28:04,258 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:28:04,271 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-54) [280339cb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:28:04,273 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [280339cb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 413f0c05 >2014-04-29 12:28:04,276 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [280339cb] FINISH, SetVdsStatusVDSCommand, log id: 413f0c05 >2014-04-29 12:28:04,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [280339cb] Correlation ID: 280339cb, Job ID: 93de4338-4312-4268-b62b-04f296524449, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:28:04,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [280339cb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:28:04,293 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [280339cb] Correlation ID: 3fa02f87, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:28:04,299 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-54) [ca14b82] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:28:04,300 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:28:04,301 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) Autorecovering 1 hosts >2014-04-29 12:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:30:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 56c10f9b >2014-04-29 12:30:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] FINISH, SetVdsStatusVDSCommand, log id: 56c10f9b >2014-04-29 12:30:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 79b2c467 >2014-04-29 12:30:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2d47b026 >2014-04-29 12:30:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] Failed in GetHardwareInfoVDS method >2014-04-29 12:30:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:30:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3d7da603 >2014-04-29 12:30:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] HostName = 10.34.27.209 >2014-04-29 12:30:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:30:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [f925aea] FINISH, GetHardwareInfoVDSCommand, log id: 2d47b026 >2014-04-29 12:30:00,697 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [f925aea] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:30:00,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [759d466b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:30:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:30:00,710 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 79b2c467 >2014-04-29 12:30:00,725 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:30:00,726 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:30:01,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 66107e2e >2014-04-29 12:30:01,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] Failed in GetHardwareInfoVDS method >2014-04-29 12:30:01,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:30:01,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@44cc325b >2014-04-29 12:30:01,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] HostName = 10.34.27.209 >2014-04-29 12:30:01,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:30:01,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [2213d596] FINISH, GetHardwareInfoVDSCommand, log id: 66107e2e >2014-04-29 12:30:01,870 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [2213d596] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:30:01,879 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [2213d596] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:30:02,097 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-34) [377de46] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:30:02,098 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [65f62408] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:30:02,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [65f62408] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 300672b3 >2014-04-29 12:33:02,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [65f62408] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:33:02,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [65f62408] FINISH, ConnectStorageServerVDSCommand, log id: 300672b3 >2014-04-29 12:33:02,106 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [65f62408] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:33:02,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 45503214 >2014-04-29 12:33:02,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 45503214 >2014-04-29 12:33:02,539 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:33:02,559 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-34) [316f87f6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:33:02,561 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [316f87f6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2cf336bb >2014-04-29 12:33:02,563 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [316f87f6] FINISH, SetVdsStatusVDSCommand, log id: 2cf336bb >2014-04-29 12:33:02,569 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [316f87f6] Correlation ID: 316f87f6, Job ID: 768c7695-04ba-446b-982c-3c2c8b8d5bd7, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:33:02,575 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [316f87f6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:33:02,577 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [316f87f6] Correlation ID: 377de46, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:33:02,580 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [4c5fa3a3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:33:02,581 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [46a2d963] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:33:02,582 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [46a2d963] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-6) Autorecovering 1 hosts >2014-04-29 12:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-6) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 41cd4a83 >2014-04-29 12:35:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] FINISH, SetVdsStatusVDSCommand, log id: 41cd4a83 >2014-04-29 12:35:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5e56a1ea >2014-04-29 12:35:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4bdce446 >2014-04-29 12:35:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] Failed in GetHardwareInfoVDS method >2014-04-29 12:35:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:35:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@130ed693 >2014-04-29 12:35:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] HostName = 10.34.27.209 >2014-04-29 12:35:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:35:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [5b886cc8] FINISH, GetHardwareInfoVDSCommand, log id: 4bdce446 >2014-04-29 12:35:00,696 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [5b886cc8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:35:00,707 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [724f74d6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:35:00,708 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:35:00,709 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5e56a1ea >2014-04-29 12:35:00,723 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:35:00,724 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:35:02,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4db7b405 >2014-04-29 12:35:03,167 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] Failed in GetHardwareInfoVDS method >2014-04-29 12:35:03,167 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:35:03,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@f5f3a69 >2014-04-29 12:35:03,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] HostName = 10.34.27.209 >2014-04-29 12:35:03,167 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:35:03,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] FINISH, GetHardwareInfoVDSCommand, log id: 4db7b405 >2014-04-29 12:35:03,173 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:35:03,182 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [6db4a9cf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:35:03,398 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-79) [22956482] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:35:03,400 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-79) [ff76ad4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:35:03,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [ff76ad4] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 35d2ec9f >2014-04-29 12:36:04,986 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-63) Start refreshing all users data >2014-04-29 12:36:07,233 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Attempting to update VMs/Templates Ovf. >2014-04-29 12:36:07,234 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Attempting to update VM OVFs in Data Center NFS >2014-04-29 12:36:07,235 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Successfully updated VM OVFs in Data Center NFS >2014-04-29 12:36:07,235 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Attempting to update template OVFs in Data Center NFS >2014-04-29 12:36:07,237 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Successfully updated templates OVFs in Data Center NFS >2014-04-29 12:36:07,237 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 12:36:07,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 12:38:03,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [ff76ad4] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:38:03,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [ff76ad4] FINISH, ConnectStorageServerVDSCommand, log id: 35d2ec9f >2014-04-29 12:38:03,408 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-79) [ff76ad4] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:38:03,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 614b3d75 >2014-04-29 12:38:03,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 614b3d75 >2014-04-29 12:38:03,843 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:38:03,857 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-79) [1f9246af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:38:03,859 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [1f9246af] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 23b11173 >2014-04-29 12:38:03,861 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [1f9246af] FINISH, SetVdsStatusVDSCommand, log id: 23b11173 >2014-04-29 12:38:03,868 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [1f9246af] Correlation ID: 1f9246af, Job ID: da6423f1-a0ae-4ad1-88a6-0f3275cb0d0e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:38:03,874 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [1f9246af] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:38:03,877 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [1f9246af] Correlation ID: 22956482, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:38:03,883 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [4297b0d2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:38:03,884 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [4031c767] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:38:03,885 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) [4031c767] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [6a0c5f21] Autorecovering 1 hosts >2014-04-29 12:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [6a0c5f21] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7d6ba4c3 >2014-04-29 12:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] FINISH, SetVdsStatusVDSCommand, log id: 7d6ba4c3 >2014-04-29 12:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: f1a63f7 >2014-04-29 12:40:00,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 70ac777 >2014-04-29 12:40:00,681 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] Failed in GetHardwareInfoVDS method >2014-04-29 12:40:00,681 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:40:00,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@54ad593d >2014-04-29 12:40:00,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] HostName = 10.34.27.209 >2014-04-29 12:40:00,681 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:40:00,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [b787f94] FINISH, GetHardwareInfoVDSCommand, log id: 70ac777 >2014-04-29 12:40:00,686 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [b787f94] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:40:00,701 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-97) [4e5842a0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:40:00,702 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-97) [7ca93cb0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:40:00,703 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) [7ca93cb0] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: f1a63f7 >2014-04-29 12:40:00,719 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [7ca93cb0] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:40:00,720 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [7ca93cb0] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:40:01,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 22fe6376 >2014-04-29 12:40:01,459 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) Failed in GetHardwareInfoVDS method >2014-04-29 12:40:01,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:40:01,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6b33e9ec >2014-04-29 12:40:01,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) HostName = 10.34.27.209 >2014-04-29 12:40:01,460 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:40:01,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) FINISH, GetHardwareInfoVDSCommand, log id: 22fe6376 >2014-04-29 12:40:01,464 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:40:01,473 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:40:01,688 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-89) [20032f0f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:40:01,689 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [d9690b6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:40:01,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [d9690b6] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 422c0589 >2014-04-29 12:43:01,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [d9690b6] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:43:01,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [d9690b6] FINISH, ConnectStorageServerVDSCommand, log id: 422c0589 >2014-04-29 12:43:01,698 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [d9690b6] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:43:01,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 44764c58 >2014-04-29 12:43:02,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 44764c58 >2014-04-29 12:43:02,133 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:43:02,147 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-89) [ca99a9b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:43:02,149 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [ca99a9b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7d410b9d >2014-04-29 12:43:02,151 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [ca99a9b] FINISH, SetVdsStatusVDSCommand, log id: 7d410b9d >2014-04-29 12:43:02,158 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [ca99a9b] Correlation ID: ca99a9b, Job ID: a103b7b4-e808-44c6-b92e-257be758e42f, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:43:02,164 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [ca99a9b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:43:02,166 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [ca99a9b] Correlation ID: 20032f0f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:43:02,170 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [297a1042] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:43:02,171 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [c79bcbe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:43:02,172 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [c79bcbe] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-91) Autorecovering 1 hosts >2014-04-29 12:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-91) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [47527356] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [47527356] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [47527356] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [47527356] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:45:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) [47527356] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 47899070 >2014-04-29 12:45:00,019 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) [47527356] FINISH, SetVdsStatusVDSCommand, log id: 47899070 >2014-04-29 12:45:00,021 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-91) [47527356] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 53fc0a1 >2014-04-29 12:45:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [47527356] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7d5954f4 >2014-04-29 12:45:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [47527356] Failed in GetHardwareInfoVDS method >2014-04-29 12:45:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [47527356] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:45:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [47527356] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@12cd3765 >2014-04-29 12:45:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [47527356] HostName = 10.34.27.209 >2014-04-29 12:45:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [47527356] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:45:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [47527356] FINISH, GetHardwareInfoVDSCommand, log id: 7d5954f4 >2014-04-29 12:45:00,697 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [47527356] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:45:00,706 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [10c505f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:45:00,707 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [196fb384] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:45:00,708 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-91) [196fb384] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 53fc0a1 >2014-04-29 12:45:00,721 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [196fb384] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:45:00,722 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [196fb384] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:45:02,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5b4cbebb >2014-04-29 12:45:02,754 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] Failed in GetHardwareInfoVDS method >2014-04-29 12:45:02,754 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:45:02,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@b7e15d1 >2014-04-29 12:45:02,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] HostName = 10.34.27.209 >2014-04-29 12:45:02,754 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:45:02,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [6317a644] FINISH, GetHardwareInfoVDSCommand, log id: 5b4cbebb >2014-04-29 12:45:02,759 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [6317a644] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:45:02,768 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [6317a644] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:45:02,984 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-87) [6d98a81] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:45:02,985 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-87) [2d37e97a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:45:02,992 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-87) [2d37e97a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6be30de9 >2014-04-29 12:48:02,994 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-87) [2d37e97a] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:48:02,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-87) [2d37e97a] FINISH, ConnectStorageServerVDSCommand, log id: 6be30de9 >2014-04-29 12:48:02,994 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-87) [2d37e97a] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:48:03,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 22f0ae02 >2014-04-29 12:48:03,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 22f0ae02 >2014-04-29 12:48:03,432 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:48:03,443 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-87) [6310a725] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:48:03,444 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [6310a725] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 251439c2 >2014-04-29 12:48:03,447 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [6310a725] FINISH, SetVdsStatusVDSCommand, log id: 251439c2 >2014-04-29 12:48:03,452 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [6310a725] Correlation ID: 6310a725, Job ID: 4893900a-5ffa-443f-8f61-5ecac42cd008, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:48:03,458 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [6310a725] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:48:03,460 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [6310a725] Correlation ID: 6d98a81, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:48:03,464 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [568a5187] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:48:03,465 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-87) [10a7c8e8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:48:03,466 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [10a7c8e8] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) Autorecovering 1 hosts >2014-04-29 12:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 744df0dc >2014-04-29 12:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] FINISH, SetVdsStatusVDSCommand, log id: 744df0dc >2014-04-29 12:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 30da5fde >2014-04-29 12:50:00,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7864f448 >2014-04-29 12:50:00,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] Failed in GetHardwareInfoVDS method >2014-04-29 12:50:00,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:50:00,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2be1e0b8 >2014-04-29 12:50:00,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] HostName = 10.34.27.209 >2014-04-29 12:50:00,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:50:00,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [5d89548a] FINISH, GetHardwareInfoVDSCommand, log id: 7864f448 >2014-04-29 12:50:00,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [5d89548a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:50:00,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-14) [4281b486] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:50:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-14) [1afcbd84] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:50:00,710 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) [1afcbd84] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 30da5fde >2014-04-29 12:50:00,725 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [1afcbd84] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:50:00,726 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [1afcbd84] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:50:03,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [56601b24] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1090b74d >2014-04-29 12:50:04,068 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [56601b24] Failed in GetHardwareInfoVDS method >2014-04-29 12:50:04,068 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [56601b24] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:50:04,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [56601b24] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@135fafe7 >2014-04-29 12:50:04,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [56601b24] HostName = 10.34.27.209 >2014-04-29 12:50:04,069 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [56601b24] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:50:04,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [56601b24] FINISH, GetHardwareInfoVDSCommand, log id: 1090b74d >2014-04-29 12:50:04,073 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [56601b24] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:50:04,081 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [56601b24] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:50:04,329 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-2) [4b75b142] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:50:04,331 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-2) [7976f6cd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:50:04,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-2) [7976f6cd] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3c7fad9d >2014-04-29 12:53:04,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-2) [7976f6cd] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:53:04,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-2) [7976f6cd] FINISH, ConnectStorageServerVDSCommand, log id: 3c7fad9d >2014-04-29 12:53:04,340 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-2) [7976f6cd] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:53:04,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 52b22977 >2014-04-29 12:53:04,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 52b22977 >2014-04-29 12:53:04,776 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:53:04,788 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-2) [583d141a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:53:04,789 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [583d141a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6e442cfd >2014-04-29 12:53:04,792 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [583d141a] FINISH, SetVdsStatusVDSCommand, log id: 6e442cfd >2014-04-29 12:53:04,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [583d141a] Correlation ID: 583d141a, Job ID: 7b1c0325-613e-42db-b1db-c7dc5de02133, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:53:04,807 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [583d141a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:53:04,810 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [583d141a] Correlation ID: 4b75b142, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:53:04,814 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-2) [40ce32a4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:53:04,814 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-2) [5fe11796] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:53:04,815 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [5fe11796] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 12:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-59) [19f188f3] Autorecovering 1 hosts >2014-04-29 12:55:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-59) [19f188f3] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 12:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 12:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 12:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2e6a3b28 >2014-04-29 12:55:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] FINISH, SetVdsStatusVDSCommand, log id: 2e6a3b28 >2014-04-29 12:55:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 47e2a7b2 >2014-04-29 12:55:00,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 320d6cf4 >2014-04-29 12:55:00,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] Failed in GetHardwareInfoVDS method >2014-04-29 12:55:00,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:55:00,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@f6c20d8 >2014-04-29 12:55:00,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] HostName = 10.34.27.209 >2014-04-29 12:55:00,807 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:55:00,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [7db39432] FINISH, GetHardwareInfoVDSCommand, log id: 320d6cf4 >2014-04-29 12:55:00,812 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [7db39432] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:55:00,820 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-59) [d8b67c7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:55:00,821 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:55:00,822 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 47e2a7b2 >2014-04-29 12:55:00,835 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 12:55:00,836 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 12:55:02,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 463577a0 >2014-04-29 12:55:02,384 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] Failed in GetHardwareInfoVDS method >2014-04-29 12:55:02,384 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:55:02,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6040cd62 >2014-04-29 12:55:02,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] HostName = 10.34.27.209 >2014-04-29 12:55:02,384 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 12:55:02,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [68b33109] FINISH, GetHardwareInfoVDSCommand, log id: 463577a0 >2014-04-29 12:55:02,389 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [68b33109] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 12:55:02,399 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [68b33109] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 12:55:02,615 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-77) [bb58ef5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:55:02,616 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [1fd28994] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 12:55:02,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [1fd28994] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6908343d >2014-04-29 12:58:02,624 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [1fd28994] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 12:58:02,624 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [1fd28994] FINISH, ConnectStorageServerVDSCommand, log id: 6908343d >2014-04-29 12:58:02,625 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [1fd28994] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 12:58:02,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: b85b381 >2014-04-29 12:58:03,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: b85b381 >2014-04-29 12:58:03,067 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 12:58:03,076 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-77) [51d760a8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:58:03,077 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [51d760a8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 434e5539 >2014-04-29 12:58:03,080 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [51d760a8] FINISH, SetVdsStatusVDSCommand, log id: 434e5539 >2014-04-29 12:58:03,085 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [51d760a8] Correlation ID: 51d760a8, Job ID: bd451f7f-c0c1-44f2-8ac5-130d3f182761, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 12:58:03,091 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [51d760a8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 12:58:03,095 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [51d760a8] Correlation ID: bb58ef5, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 12:58:03,099 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [b7afdec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:58:03,101 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [2b34b0c9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 12:58:03,101 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [2b34b0c9] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-69) [273b756f] Autorecovering 1 hosts >2014-04-29 13:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-69) [273b756f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7fd804c6 >2014-04-29 13:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] FINISH, SetVdsStatusVDSCommand, log id: 7fd804c6 >2014-04-29 13:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 64632d10 >2014-04-29 13:00:00,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 212ccb37 >2014-04-29 13:00:00,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] Failed in GetHardwareInfoVDS method >2014-04-29 13:00:00,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:00:00,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2bbc5a20 >2014-04-29 13:00:00,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] HostName = 10.34.27.209 >2014-04-29 13:00:00,684 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:00:00,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7287cec] FINISH, GetHardwareInfoVDSCommand, log id: 212ccb37 >2014-04-29 13:00:00,689 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [7287cec] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:00:00,700 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-69) [1b4ea13b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:00:00,701 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:00:00,701 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 64632d10 >2014-04-29 13:00:00,719 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:00:00,723 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:00:03,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [21697a9d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7840c87c >2014-04-29 13:00:03,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [21697a9d] Failed in GetHardwareInfoVDS method >2014-04-29 13:00:03,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [21697a9d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:00:03,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [21697a9d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5fbea5af >2014-04-29 13:00:03,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [21697a9d] HostName = 10.34.27.209 >2014-04-29 13:00:03,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [21697a9d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:00:03,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [21697a9d] FINISH, GetHardwareInfoVDSCommand, log id: 7840c87c >2014-04-29 13:00:03,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [21697a9d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:00:03,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [21697a9d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:00:03,922 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-56) [215cbec2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:00:03,924 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-56) [5f6c5b7b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:00:03,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [5f6c5b7b] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6027316 >2014-04-29 13:03:03,932 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [5f6c5b7b] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:03:03,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [5f6c5b7b] FINISH, ConnectStorageServerVDSCommand, log id: 6027316 >2014-04-29 13:03:03,932 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-56) [5f6c5b7b] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:03:03,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 14c1cb2c >2014-04-29 13:03:04,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 14c1cb2c >2014-04-29 13:03:04,393 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:03:04,509 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-56) [30bd162] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:03:04,510 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [30bd162] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7ace11f9 >2014-04-29 13:03:04,539 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [30bd162] FINISH, SetVdsStatusVDSCommand, log id: 7ace11f9 >2014-04-29 13:03:04,568 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [30bd162] Correlation ID: 30bd162, Job ID: 13028958-2c80-4b8a-8a96-9bb028495433, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:03:04,602 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [30bd162] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:03:04,628 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [30bd162] Correlation ID: 215cbec2, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:03:04,681 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-56) [41d27c5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:03:04,682 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-56) [30349ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:03:04,683 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [30349ac] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-49) [6ed4d8d8] Autorecovering 1 hosts >2014-04-29 13:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-49) [6ed4d8d8] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1ccb068 >2014-04-29 13:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] FINISH, SetVdsStatusVDSCommand, log id: 1ccb068 >2014-04-29 13:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 19a1499a >2014-04-29 13:05:00,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 104b41f3 >2014-04-29 13:05:00,681 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] Failed in GetHardwareInfoVDS method >2014-04-29 13:05:00,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:05:00,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3cd6c0e0 >2014-04-29 13:05:00,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] HostName = 10.34.27.209 >2014-04-29 13:05:00,682 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:05:00,682 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [4394a65] FINISH, GetHardwareInfoVDSCommand, log id: 104b41f3 >2014-04-29 13:05:00,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [4394a65] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:05:00,699 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-49) [3886b608] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:05:00,701 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-49) [34373ba8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:05:00,702 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [34373ba8] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 19a1499a >2014-04-29 13:05:00,723 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [34373ba8] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:05:00,724 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [34373ba8] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:05:02,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6be23687 >2014-04-29 13:05:02,259 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] Failed in GetHardwareInfoVDS method >2014-04-29 13:05:02,259 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:05:02,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@685ce0b6 >2014-04-29 13:05:02,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] HostName = 10.34.27.209 >2014-04-29 13:05:02,259 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:05:02,259 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] FINISH, GetHardwareInfoVDSCommand, log id: 6be23687 >2014-04-29 13:05:02,264 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:05:02,272 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [65d3d4f1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:05:02,503 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-53) [8712390] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:05:02,505 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-53) [73658d01] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:05:02,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [73658d01] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 33fb462 >2014-04-29 13:08:02,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [73658d01] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:08:02,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [73658d01] FINISH, ConnectStorageServerVDSCommand, log id: 33fb462 >2014-04-29 13:08:02,513 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-53) [73658d01] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:08:02,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: ec4737a >2014-04-29 13:08:02,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: ec4737a >2014-04-29 13:08:02,962 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:08:02,972 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-53) [71ac9189] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:08:02,973 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [71ac9189] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 53a31764 >2014-04-29 13:08:02,975 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [71ac9189] FINISH, SetVdsStatusVDSCommand, log id: 53a31764 >2014-04-29 13:08:02,982 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [71ac9189] Correlation ID: 71ac9189, Job ID: 2642f282-8389-4656-b17b-2ff5655849bd, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:08:02,988 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [71ac9189] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:08:02,990 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [71ac9189] Correlation ID: 8712390, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:08:02,994 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-53) [6e88e39f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:08:02,995 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-53) [353e2b4d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:08:02,996 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) [353e2b4d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) Autorecovering 1 hosts >2014-04-29 13:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: cae8148 >2014-04-29 13:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] FINISH, SetVdsStatusVDSCommand, log id: cae8148 >2014-04-29 13:10:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 607c7edb >2014-04-29 13:10:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2c100b04 >2014-04-29 13:10:00,929 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] Failed in GetHardwareInfoVDS method >2014-04-29 13:10:00,929 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:10:00,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@51434a35 >2014-04-29 13:10:00,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] HostName = 10.34.27.209 >2014-04-29 13:10:00,929 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:10:00,929 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [71852c64] FINISH, GetHardwareInfoVDSCommand, log id: 2c100b04 >2014-04-29 13:10:00,934 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [71852c64] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:10:00,944 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [32f5de7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:10:00,946 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:10:00,946 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 607c7edb >2014-04-29 13:10:00,964 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:10:00,965 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:10:03,346 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: b9584ee >2014-04-29 13:10:03,563 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] Failed in GetHardwareInfoVDS method >2014-04-29 13:10:03,563 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:10:03,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@54a2b33 >2014-04-29 13:10:03,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] HostName = 10.34.27.209 >2014-04-29 13:10:03,563 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:10:03,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] FINISH, GetHardwareInfoVDSCommand, log id: b9584ee >2014-04-29 13:10:03,569 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:10:03,577 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [4e2c37eb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:10:03,809 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-31) [350d6459] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:10:03,810 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [3f31f786] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:10:03,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [3f31f786] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 60cef991 >2014-04-29 13:13:03,818 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [3f31f786] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:13:03,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [3f31f786] FINISH, ConnectStorageServerVDSCommand, log id: 60cef991 >2014-04-29 13:13:03,818 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [3f31f786] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:13:03,822 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7ee953fc >2014-04-29 13:13:04,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7ee953fc >2014-04-29 13:13:04,254 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:13:04,270 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-31) [6f72d27] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:13:04,272 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [6f72d27] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2523b721 >2014-04-29 13:13:04,274 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [6f72d27] FINISH, SetVdsStatusVDSCommand, log id: 2523b721 >2014-04-29 13:13:04,280 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [6f72d27] Correlation ID: 6f72d27, Job ID: 68da5fa8-d34e-4471-9e87-1c05faf95753, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:13:04,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [6f72d27] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:13:04,287 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [6f72d27] Correlation ID: 350d6459, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:13:04,291 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-31) [4ee8cc65] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:13:04,292 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-31) [65605de0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:13:04,293 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [65605de0] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:15:00,012 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) [6af5af0a] Autorecovering 1 hosts >2014-04-29 13:15:00,012 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) [6af5af0a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:15:00,013 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:15:00,014 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:15:00,014 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:15:00,014 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:15:00,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 9266feb >2014-04-29 13:15:00,020 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] FINISH, SetVdsStatusVDSCommand, log id: 9266feb >2014-04-29 13:15:00,021 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5268cb49 >2014-04-29 13:15:00,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 37ccb56b >2014-04-29 13:15:00,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] Failed in GetHardwareInfoVDS method >2014-04-29 13:15:00,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:15:00,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@609aa06b >2014-04-29 13:15:00,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] HostName = 10.34.27.209 >2014-04-29 13:15:00,964 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:15:00,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [4d9a9143] FINISH, GetHardwareInfoVDSCommand, log id: 37ccb56b >2014-04-29 13:15:00,995 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [4d9a9143] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:15:01,076 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [79098224] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:15:01,077 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:15:01,078 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5268cb49 >2014-04-29 13:15:01,113 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:15:01,114 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:15:01,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1655e40b >2014-04-29 13:15:01,882 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Failed in GetHardwareInfoVDS method >2014-04-29 13:15:01,882 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:15:01,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@343a8240 >2014-04-29 13:15:01,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] HostName = 10.34.27.209 >2014-04-29 13:15:01,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:15:01,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] FINISH, GetHardwareInfoVDSCommand, log id: 1655e40b >2014-04-29 13:15:01,889 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:15:01,897 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [6d97cd9c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:15:02,122 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-55) [41cb834b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:15:02,124 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [3c19c6cf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:15:02,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [3c19c6cf] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 48a50223 >2014-04-29 13:18:02,132 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [3c19c6cf] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:18:02,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [3c19c6cf] FINISH, ConnectStorageServerVDSCommand, log id: 48a50223 >2014-04-29 13:18:02,132 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [3c19c6cf] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:18:02,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 38a85c5f >2014-04-29 13:18:02,571 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 38a85c5f >2014-04-29 13:18:02,571 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:18:02,604 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-55) [1f1b08e6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:18:02,606 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [1f1b08e6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7437ac32 >2014-04-29 13:18:02,614 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [1f1b08e6] FINISH, SetVdsStatusVDSCommand, log id: 7437ac32 >2014-04-29 13:18:02,623 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [1f1b08e6] Correlation ID: 1f1b08e6, Job ID: b3f11ba3-6fb7-4998-9f80-296c081db2e3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:18:02,636 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [1f1b08e6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:18:02,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [1f1b08e6] Correlation ID: 41cb834b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:18:02,645 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [7be43c4e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:18:02,647 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [e3e34a3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:18:02,647 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [e3e34a3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-22) [63759f74] Autorecovering 1 hosts >2014-04-29 13:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-22) [63759f74] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4d14d93 >2014-04-29 13:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] FINISH, SetVdsStatusVDSCommand, log id: 4d14d93 >2014-04-29 13:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 43d763aa >2014-04-29 13:20:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5fb08afc >2014-04-29 13:20:00,915 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] Failed in GetHardwareInfoVDS method >2014-04-29 13:20:00,915 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:20:00,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3f74d778 >2014-04-29 13:20:00,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] HostName = 10.34.27.209 >2014-04-29 13:20:00,919 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:20:00,919 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [4b146afe] FINISH, GetHardwareInfoVDSCommand, log id: 5fb08afc >2014-04-29 13:20:00,924 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [4b146afe] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:20:00,932 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-22) [6491287d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:20:00,934 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:20:00,934 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 43d763aa >2014-04-29 13:20:00,954 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:20:00,955 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:20:03,019 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 27f55eb7 >2014-04-29 13:20:03,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) Failed in GetHardwareInfoVDS method >2014-04-29 13:20:03,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:20:03,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5e1d54df >2014-04-29 13:20:03,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) HostName = 10.34.27.209 >2014-04-29 13:20:03,252 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:20:03,252 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) FINISH, GetHardwareInfoVDSCommand, log id: 27f55eb7 >2014-04-29 13:20:03,257 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:20:03,266 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:20:03,494 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-1) [4052724d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:20:03,495 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-1) [d7ecf54] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:20:03,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-1) [d7ecf54] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 74cedbf1 >2014-04-29 13:23:03,504 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-1) [d7ecf54] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:23:03,504 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-1) [d7ecf54] FINISH, ConnectStorageServerVDSCommand, log id: 74cedbf1 >2014-04-29 13:23:03,504 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-1) [d7ecf54] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:23:03,508 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 449295ad >2014-04-29 13:23:03,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 449295ad >2014-04-29 13:23:03,939 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:23:03,950 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-1) [43d51400] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:23:03,951 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-1) [43d51400] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 9be1297 >2014-04-29 13:23:03,954 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-1) [43d51400] FINISH, SetVdsStatusVDSCommand, log id: 9be1297 >2014-04-29 13:23:03,959 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) [43d51400] Correlation ID: 43d51400, Job ID: 33dc3380-536f-4a70-87de-0f9e72d72dac, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:23:03,966 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) [43d51400] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:23:03,968 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) [43d51400] Correlation ID: 4052724d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:23:03,971 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-1) [4c8ebc3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:23:03,972 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-1) [730c095] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:23:03,973 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-1) [730c095] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-9) Autorecovering 1 hosts >2014-04-29 13:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-9) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 67f13b8b >2014-04-29 13:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] FINISH, SetVdsStatusVDSCommand, log id: 67f13b8b >2014-04-29 13:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2bb08abd >2014-04-29 13:25:00,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6d31e3c1 >2014-04-29 13:25:00,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] Failed in GetHardwareInfoVDS method >2014-04-29 13:25:00,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:25:00,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@665fcbe8 >2014-04-29 13:25:00,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] HostName = 10.34.27.209 >2014-04-29 13:25:00,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:25:00,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [601c5803] FINISH, GetHardwareInfoVDSCommand, log id: 6d31e3c1 >2014-04-29 13:25:00,861 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [601c5803] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:25:00,869 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [5896a440] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:25:00,870 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [624b825c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:25:00,871 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [624b825c] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2bb08abd >2014-04-29 13:25:00,884 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [624b825c] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:25:00,885 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [624b825c] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:25:01,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [3641f495] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6965b946 >2014-04-29 13:25:01,554 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [3641f495] Failed in GetHardwareInfoVDS method >2014-04-29 13:25:01,554 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [3641f495] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:25:01,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [3641f495] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1566d8b9 >2014-04-29 13:25:01,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [3641f495] HostName = 10.34.27.209 >2014-04-29 13:25:01,554 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [3641f495] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:25:01,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [3641f495] FINISH, GetHardwareInfoVDSCommand, log id: 6965b946 >2014-04-29 13:25:01,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [3641f495] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:25:01,569 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [3641f495] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:25:01,799 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-43) [1815bb97] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:25:01,804 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [157919ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:25:01,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [157919ed] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 34ba9a45 >2014-04-29 13:28:01,811 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [157919ed] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:28:01,811 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [157919ed] FINISH, ConnectStorageServerVDSCommand, log id: 34ba9a45 >2014-04-29 13:28:01,811 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [157919ed] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:28:01,818 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 56013953 >2014-04-29 13:28:02,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 56013953 >2014-04-29 13:28:02,257 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:28:02,270 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-43) [41a1e4b1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:28:02,271 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [41a1e4b1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1c43cc95 >2014-04-29 13:28:02,274 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [41a1e4b1] FINISH, SetVdsStatusVDSCommand, log id: 1c43cc95 >2014-04-29 13:28:02,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [41a1e4b1] Correlation ID: 41a1e4b1, Job ID: 4f07d3b1-98c3-4cf2-84d6-283ee5d49132, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:28:02,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [41a1e4b1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:28:02,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [41a1e4b1] Correlation ID: 1815bb97, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:28:02,296 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [2400d584] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:28:02,297 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [60fc8501] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:28:02,298 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [60fc8501] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-82) [24ac60e8] Autorecovering 1 hosts >2014-04-29 13:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-82) [24ac60e8] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [48454783] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [48454783] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [48454783] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [48454783] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [48454783] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 20c31bee >2014-04-29 13:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [48454783] FINISH, SetVdsStatusVDSCommand, log id: 20c31bee >2014-04-29 13:30:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) [48454783] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: c367da3 >2014-04-29 13:30:00,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [48454783] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1ea464cd >2014-04-29 13:30:00,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [48454783] Failed in GetHardwareInfoVDS method >2014-04-29 13:30:00,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [48454783] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:30:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [48454783] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3ad5de72 >2014-04-29 13:30:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [48454783] HostName = 10.34.27.209 >2014-04-29 13:30:00,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [48454783] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:30:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [48454783] FINISH, GetHardwareInfoVDSCommand, log id: 1ea464cd >2014-04-29 13:30:00,919 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [48454783] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:30:00,932 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-82) [55635b08] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:30:00,934 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:30:00,934 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: c367da3 >2014-04-29 13:30:00,948 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:30:00,949 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:30:02,662 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5f70e517 >2014-04-29 13:30:02,880 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] Failed in GetHardwareInfoVDS method >2014-04-29 13:30:02,880 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:30:02,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6cd079f4 >2014-04-29 13:30:02,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] HostName = 10.34.27.209 >2014-04-29 13:30:02,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:30:02,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [5367ff41] FINISH, GetHardwareInfoVDSCommand, log id: 5f70e517 >2014-04-29 13:30:02,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [5367ff41] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:30:02,900 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [5367ff41] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:30:03,127 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-40) [4b369c9f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:30:03,128 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-40) [d992b91] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:30:03,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-40) [d992b91] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 585a31d2 >2014-04-29 13:33:03,136 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-40) [d992b91] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:33:03,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-40) [d992b91] FINISH, ConnectStorageServerVDSCommand, log id: 585a31d2 >2014-04-29 13:33:03,136 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-40) [d992b91] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:33:03,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7a9738e5 >2014-04-29 13:33:03,583 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7a9738e5 >2014-04-29 13:33:03,583 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:33:03,615 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-40) [7b4347e2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:33:03,616 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [7b4347e2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4589fe81 >2014-04-29 13:33:03,621 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [7b4347e2] FINISH, SetVdsStatusVDSCommand, log id: 4589fe81 >2014-04-29 13:33:03,627 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [7b4347e2] Correlation ID: 7b4347e2, Job ID: 2750088e-9c1f-4f4b-9295-201b24997438, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:33:03,637 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [7b4347e2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:33:03,641 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [7b4347e2] Correlation ID: 4b369c9f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:33:03,650 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-40) [78119ede] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:33:03,652 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-40) [5c5e48bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:33:03,652 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) [5c5e48bc] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-5) Autorecovering 1 hosts >2014-04-29 13:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-5) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 77774f03 >2014-04-29 13:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] FINISH, SetVdsStatusVDSCommand, log id: 77774f03 >2014-04-29 13:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3eb4d03d >2014-04-29 13:35:00,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 414a25c4 >2014-04-29 13:35:00,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] Failed in GetHardwareInfoVDS method >2014-04-29 13:35:00,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:35:00,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6af58858 >2014-04-29 13:35:00,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] HostName = 10.34.27.209 >2014-04-29 13:35:00,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:35:00,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [40177d8b] FINISH, GetHardwareInfoVDSCommand, log id: 414a25c4 >2014-04-29 13:35:00,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [40177d8b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:35:00,898 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-5) [7b9838ae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:35:00,899 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-5) [21573f3a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:35:00,900 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) [21573f3a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3eb4d03d >2014-04-29 13:35:00,918 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [21573f3a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:35:00,919 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [21573f3a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:35:04,008 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 676183f4 >2014-04-29 13:35:04,233 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] Failed in GetHardwareInfoVDS method >2014-04-29 13:35:04,233 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:35:04,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@331e3d32 >2014-04-29 13:35:04,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] HostName = 10.34.27.209 >2014-04-29 13:35:04,233 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:35:04,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [362cf2bf] FINISH, GetHardwareInfoVDSCommand, log id: 676183f4 >2014-04-29 13:35:04,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [362cf2bf] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:35:04,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [362cf2bf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:35:04,464 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-83) [1acdc260] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:35:04,466 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-83) [4c90100f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:35:04,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-83) [4c90100f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 51c11a98 >2014-04-29 13:36:04,989 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-60) [29a144e6] Start refreshing all users data >2014-04-29 13:36:07,238 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Attempting to update VMs/Templates Ovf. >2014-04-29 13:36:07,239 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Attempting to update VM OVFs in Data Center NFS >2014-04-29 13:36:07,244 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Successfully updated VM OVFs in Data Center NFS >2014-04-29 13:36:07,244 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Attempting to update template OVFs in Data Center NFS >2014-04-29 13:36:07,244 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Successfully updated templates OVFs in Data Center NFS >2014-04-29 13:36:07,244 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 13:36:07,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 13:38:04,475 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-83) [4c90100f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:38:04,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-83) [4c90100f] FINISH, ConnectStorageServerVDSCommand, log id: 51c11a98 >2014-04-29 13:38:04,475 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-83) [4c90100f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:38:04,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4ceed793 >2014-04-29 13:38:04,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4ceed793 >2014-04-29 13:38:04,922 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:38:04,932 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-83) [2d824cd9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:38:04,933 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [2d824cd9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 288aac90 >2014-04-29 13:38:04,936 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [2d824cd9] FINISH, SetVdsStatusVDSCommand, log id: 288aac90 >2014-04-29 13:38:04,945 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [2d824cd9] Correlation ID: 2d824cd9, Job ID: 0c492612-e4a0-4a60-ad2e-56ccddedb53b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:38:04,951 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [2d824cd9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:38:04,953 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [2d824cd9] Correlation ID: 1acdc260, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:38:04,957 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-83) [5b0c2882] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:38:04,958 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-83) [43910787] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:38:04,959 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [43910787] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) Autorecovering 1 hosts >2014-04-29 13:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2d0cd6a0 >2014-04-29 13:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] FINISH, SetVdsStatusVDSCommand, log id: 2d0cd6a0 >2014-04-29 13:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 48d2b79f >2014-04-29 13:40:01,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2cbaf3bb >2014-04-29 13:40:01,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] Failed in GetHardwareInfoVDS method >2014-04-29 13:40:01,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:40:01,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@25d790ee >2014-04-29 13:40:01,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] HostName = 10.34.27.209 >2014-04-29 13:40:01,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:40:01,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [2c07dac1] FINISH, GetHardwareInfoVDSCommand, log id: 2cbaf3bb >2014-04-29 13:40:01,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [2c07dac1] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:40:01,828 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-90) [246cb8e8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:40:01,829 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:40:01,829 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 48d2b79f >2014-04-29 13:40:01,852 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:40:01,854 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:40:02,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6d89357f >2014-04-29 13:40:03,179 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] Failed in GetHardwareInfoVDS method >2014-04-29 13:40:03,179 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:40:03,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4b175d07 >2014-04-29 13:40:03,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] HostName = 10.34.27.209 >2014-04-29 13:40:03,180 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:40:03,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [30a08bd3] FINISH, GetHardwareInfoVDSCommand, log id: 6d89357f >2014-04-29 13:40:03,186 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [30a08bd3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:40:03,195 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [30a08bd3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:40:03,737 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-50) [6f2c2cdb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:40:03,738 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [55a938ac] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:40:03,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [55a938ac] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b09d21d >2014-04-29 13:43:03,746 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [55a938ac] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:43:03,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [55a938ac] FINISH, ConnectStorageServerVDSCommand, log id: 7b09d21d >2014-04-29 13:43:03,746 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [55a938ac] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:43:03,750 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5f03ee16 >2014-04-29 13:43:04,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5f03ee16 >2014-04-29 13:43:04,179 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:43:04,190 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-50) [42f04414] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:43:04,191 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [42f04414] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 105ae767 >2014-04-29 13:43:04,193 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [42f04414] FINISH, SetVdsStatusVDSCommand, log id: 105ae767 >2014-04-29 13:43:04,201 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [42f04414] Correlation ID: 42f04414, Job ID: e67f7c2b-6c8d-4089-adff-7add1eb8938d, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:43:04,207 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [42f04414] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:43:04,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [42f04414] Correlation ID: 6f2c2cdb, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:43:04,214 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [74a1f587] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:43:04,215 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [1e05f782] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:43:04,216 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [1e05f782] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [30349ac] Autorecovering 1 hosts >2014-04-29 13:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [30349ac] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:45:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7ca815e3 >2014-04-29 13:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] FINISH, SetVdsStatusVDSCommand, log id: 7ca815e3 >2014-04-29 13:45:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1dd6212d >2014-04-29 13:45:00,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 532aee8c >2014-04-29 13:45:00,858 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] Failed in GetHardwareInfoVDS method >2014-04-29 13:45:00,858 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:45:00,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@279353fc >2014-04-29 13:45:00,858 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] HostName = 10.34.27.209 >2014-04-29 13:45:00,858 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:45:00,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [78e3c857] FINISH, GetHardwareInfoVDSCommand, log id: 532aee8c >2014-04-29 13:45:00,866 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [78e3c857] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:45:00,887 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-56) [6ced069b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:45:00,889 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-56) [76bed1f1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:45:00,890 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [76bed1f1] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1dd6212d >2014-04-29 13:45:00,907 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [76bed1f1] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:45:00,908 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [76bed1f1] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:45:01,567 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [377fdfbe] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: c3d03c3 >2014-04-29 13:45:01,783 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [377fdfbe] Failed in GetHardwareInfoVDS method >2014-04-29 13:45:01,783 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [377fdfbe] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:45:01,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [377fdfbe] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2ee8d4b0 >2014-04-29 13:45:01,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [377fdfbe] HostName = 10.34.27.209 >2014-04-29 13:45:01,783 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [377fdfbe] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:45:01,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [377fdfbe] FINISH, GetHardwareInfoVDSCommand, log id: c3d03c3 >2014-04-29 13:45:01,788 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [377fdfbe] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:45:01,804 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [377fdfbe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:45:02,020 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-52) [5bfb9dca] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:45:02,021 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-52) [318f850] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:45:02,030 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [318f850] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 18891f03 >2014-04-29 13:48:02,031 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [318f850] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:48:02,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [318f850] FINISH, ConnectStorageServerVDSCommand, log id: 18891f03 >2014-04-29 13:48:02,031 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-52) [318f850] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:48:02,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 165e592c >2014-04-29 13:48:02,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 165e592c >2014-04-29 13:48:02,454 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:48:02,471 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-52) [7f380b26] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:48:02,472 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f380b26] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 28727c3e >2014-04-29 13:48:02,475 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [7f380b26] FINISH, SetVdsStatusVDSCommand, log id: 28727c3e >2014-04-29 13:48:02,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [7f380b26] Correlation ID: 7f380b26, Job ID: 1132e145-f382-4623-8143-5a81498ece7e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:48:02,490 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [7f380b26] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:48:02,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [7f380b26] Correlation ID: 5bfb9dca, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:48:02,496 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-52) [b74ee46] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:48:02,497 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-52) [35198fe3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:48:02,498 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [35198fe3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-44) [8434417] Autorecovering 1 hosts >2014-04-29 13:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-44) [8434417] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 63288df2 >2014-04-29 13:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] FINISH, SetVdsStatusVDSCommand, log id: 63288df2 >2014-04-29 13:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3f213d0c >2014-04-29 13:50:00,623 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4e621ec7 >2014-04-29 13:50:00,853 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] Failed in GetHardwareInfoVDS method >2014-04-29 13:50:00,853 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:50:00,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@704c632d >2014-04-29 13:50:00,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] HostName = 10.34.27.209 >2014-04-29 13:50:00,853 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:50:00,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1820cc21] FINISH, GetHardwareInfoVDSCommand, log id: 4e621ec7 >2014-04-29 13:50:00,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [1820cc21] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:50:00,876 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [3d8c4132] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:50:00,878 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [34a5a5ec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:50:00,879 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) [34a5a5ec] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3f213d0c >2014-04-29 13:50:00,898 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [34a5a5ec] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:50:00,899 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [34a5a5ec] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:50:02,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5ddefd9a >2014-04-29 13:50:03,086 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) Failed in GetHardwareInfoVDS method >2014-04-29 13:50:03,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:50:03,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@f769db7 >2014-04-29 13:50:03,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) HostName = 10.34.27.209 >2014-04-29 13:50:03,087 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:50:03,087 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) FINISH, GetHardwareInfoVDSCommand, log id: 5ddefd9a >2014-04-29 13:50:03,091 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:50:03,101 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:50:03,313 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-98) [c5107d5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:50:03,315 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [7c7416fe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:50:03,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [7c7416fe] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5248b3 >2014-04-29 13:53:03,323 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [7c7416fe] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:53:03,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [7c7416fe] FINISH, ConnectStorageServerVDSCommand, log id: 5248b3 >2014-04-29 13:53:03,323 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [7c7416fe] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:53:03,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 64496c1b >2014-04-29 13:53:03,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 64496c1b >2014-04-29 13:53:03,746 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:53:03,758 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-98) [20331ec5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:53:03,759 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [20331ec5] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3c67ded3 >2014-04-29 13:53:03,761 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [20331ec5] FINISH, SetVdsStatusVDSCommand, log id: 3c67ded3 >2014-04-29 13:53:03,767 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [20331ec5] Correlation ID: 20331ec5, Job ID: a116b7d3-8be0-4d74-8fe6-613363a2f4b7, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:53:03,774 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [20331ec5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:53:03,777 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [20331ec5] Correlation ID: c5107d5, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:53:03,782 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-98) [17ab4751] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:53:03,784 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-98) [3a91960d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:53:03,784 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [3a91960d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 13:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-73) [2d7a8841] Autorecovering 1 hosts >2014-04-29 13:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-73) [2d7a8841] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 13:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 13:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 13:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5504af10 >2014-04-29 13:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] FINISH, SetVdsStatusVDSCommand, log id: 5504af10 >2014-04-29 13:55:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: adc336d >2014-04-29 13:55:00,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7dcf7732 >2014-04-29 13:55:05,997 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] Failed in GetHardwareInfoVDS method >2014-04-29 13:55:05,997 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:55:05,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2390c6eb >2014-04-29 13:55:05,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] HostName = 10.34.27.209 >2014-04-29 13:55:05,998 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:55:05,998 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [a155e3f] FINISH, GetHardwareInfoVDSCommand, log id: 7dcf7732 >2014-04-29 13:55:06,003 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [a155e3f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:55:06,012 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [51f4e64b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:55:06,013 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [639730bf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:55:06,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) [639730bf] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: adc336d >2014-04-29 13:55:06,031 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [639730bf] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 13:55:06,032 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [639730bf] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 13:55:07,140 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [44c9639a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 44348b0 >2014-04-29 13:55:07,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [44c9639a] Failed in GetHardwareInfoVDS method >2014-04-29 13:55:07,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [44c9639a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:55:07,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [44c9639a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5d591cff >2014-04-29 13:55:07,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [44c9639a] HostName = 10.34.27.209 >2014-04-29 13:55:07,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [44c9639a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 13:55:07,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [44c9639a] FINISH, GetHardwareInfoVDSCommand, log id: 44348b0 >2014-04-29 13:55:07,353 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [44c9639a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 13:55:07,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [44c9639a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 13:55:07,573 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-47) [28aaf729] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:55:07,575 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-47) [7110103d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 13:55:07,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-47) [7110103d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6dd7a5d3 >2014-04-29 13:58:07,582 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-47) [7110103d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 13:58:07,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-47) [7110103d] FINISH, ConnectStorageServerVDSCommand, log id: 6dd7a5d3 >2014-04-29 13:58:07,582 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-47) [7110103d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 13:58:07,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 13afe253 >2014-04-29 13:58:08,012 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 13afe253 >2014-04-29 13:58:08,012 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 13:58:08,030 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-47) [7c7e779e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:58:08,031 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [7c7e779e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4ff69803 >2014-04-29 13:58:08,034 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [7c7e779e] FINISH, SetVdsStatusVDSCommand, log id: 4ff69803 >2014-04-29 13:58:08,040 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [7c7e779e] Correlation ID: 7c7e779e, Job ID: 8c9da3ee-3e80-4e6b-bffd-5ea138fff916, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 13:58:08,047 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [7c7e779e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 13:58:08,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [7c7e779e] Correlation ID: 28aaf729, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 13:58:08,058 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-47) [528c5c1a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:58:08,063 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-47) [4796b052] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 13:58:08,067 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [4796b052] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) [1afcbd84] Autorecovering 1 hosts >2014-04-29 14:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) [1afcbd84] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7995eeb4 >2014-04-29 14:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] FINISH, SetVdsStatusVDSCommand, log id: 7995eeb4 >2014-04-29 14:00:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 72c8cc3f >2014-04-29 14:00:00,619 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 28edcef2 >2014-04-29 14:00:00,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] Failed in GetHardwareInfoVDS method >2014-04-29 14:00:00,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:00:00,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@14ad86dc >2014-04-29 14:00:00,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] HostName = 10.34.27.209 >2014-04-29 14:00:00,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:00:00,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [c2fbb48] FINISH, GetHardwareInfoVDSCommand, log id: 28edcef2 >2014-04-29 14:00:00,841 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [c2fbb48] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:00:00,856 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-14) [1b7a60ab] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:00:00,857 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-14) [621d3184] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:00:00,858 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) [621d3184] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 72c8cc3f >2014-04-29 14:00:00,871 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [621d3184] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:00:00,872 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [621d3184] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:00:02,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6f0a3888 >2014-04-29 14:00:02,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] Failed in GetHardwareInfoVDS method >2014-04-29 14:00:02,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:00:02,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@805a1ac >2014-04-29 14:00:02,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] HostName = 10.34.27.209 >2014-04-29 14:00:02,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:00:02,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [61a731bd] FINISH, GetHardwareInfoVDSCommand, log id: 6f0a3888 >2014-04-29 14:00:02,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [61a731bd] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:00:02,651 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [61a731bd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:00:02,861 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-78) [696c1ac4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:00:02,863 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [cd8f280] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:00:02,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [cd8f280] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 270d1858 >2014-04-29 14:03:02,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [cd8f280] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:03:02,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [cd8f280] FINISH, ConnectStorageServerVDSCommand, log id: 270d1858 >2014-04-29 14:03:02,871 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [cd8f280] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:03:02,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 308e2ac >2014-04-29 14:03:03,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 308e2ac >2014-04-29 14:03:03,323 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:03:03,333 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-78) [4a2e116f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:03:03,334 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [4a2e116f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 21529cfd >2014-04-29 14:03:03,337 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [4a2e116f] FINISH, SetVdsStatusVDSCommand, log id: 21529cfd >2014-04-29 14:03:03,344 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4a2e116f] Correlation ID: 4a2e116f, Job ID: 2a04ec0d-5b84-4d13-8bee-149e48b13e80, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:03:03,350 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4a2e116f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:03:03,352 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4a2e116f] Correlation ID: 696c1ac4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:03:03,356 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [6af7b0ce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:03:03,357 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [7a23254] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:03:03,358 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [7a23254] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:05:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-34) [46a2d963] Autorecovering 1 hosts >2014-04-29 14:05:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-34) [46a2d963] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:05:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:05:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:05:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:05:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6983b575 >2014-04-29 14:05:00,035 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] FINISH, SetVdsStatusVDSCommand, log id: 6983b575 >2014-04-29 14:05:00,037 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 50070f30 >2014-04-29 14:05:00,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3810fea7 >2014-04-29 14:05:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] Failed in GetHardwareInfoVDS method >2014-04-29 14:05:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:05:00,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@78ab8746 >2014-04-29 14:05:00,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] HostName = 10.34.27.209 >2014-04-29 14:05:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:05:00,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [1c49a33e] FINISH, GetHardwareInfoVDSCommand, log id: 3810fea7 >2014-04-29 14:05:00,893 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [1c49a33e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:05:00,931 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [26a3415a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:05:00,933 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:05:00,935 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 50070f30 >2014-04-29 14:05:00,954 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:05:00,955 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:05:03,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 680c1f1 >2014-04-29 14:05:03,915 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) Failed in GetHardwareInfoVDS method >2014-04-29 14:05:03,915 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:05:03,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3cffab62 >2014-04-29 14:05:03,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) HostName = 10.34.27.209 >2014-04-29 14:05:03,915 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:05:03,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) FINISH, GetHardwareInfoVDSCommand, log id: 680c1f1 >2014-04-29 14:05:03,919 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:05:03,928 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:05:04,138 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-51) [172552a3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:05:04,141 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [2b99e4b1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:05:04,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [2b99e4b1] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 76017e7d >2014-04-29 14:08:04,149 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [2b99e4b1] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:08:04,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [2b99e4b1] FINISH, ConnectStorageServerVDSCommand, log id: 76017e7d >2014-04-29 14:08:04,149 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [2b99e4b1] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:08:04,153 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 73360aeb >2014-04-29 14:08:04,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 73360aeb >2014-04-29 14:08:04,582 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:08:04,594 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-51) [6daa1979] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:08:04,595 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [6daa1979] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5021afc4 >2014-04-29 14:08:04,597 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [6daa1979] FINISH, SetVdsStatusVDSCommand, log id: 5021afc4 >2014-04-29 14:08:04,603 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [6daa1979] Correlation ID: 6daa1979, Job ID: b6a409a1-50ba-40d2-bc4d-131ff0f08ed5, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:08:04,609 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [6daa1979] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:08:04,611 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [6daa1979] Correlation ID: 172552a3, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:08:04,614 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-51) [1f7f8828] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:08:04,616 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-51) [8088d97] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:08:04,616 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) [8088d97] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:10:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Autorecovering 1 hosts >2014-04-29 14:10:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) [6d63ed15] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:10:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:10:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:10:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:10:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2a1f1fac >2014-04-29 14:10:00,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] FINISH, SetVdsStatusVDSCommand, log id: 2a1f1fac >2014-04-29 14:10:00,020 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: d6a1f79 >2014-04-29 14:10:00,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 729378b6 >2014-04-29 14:10:00,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] Failed in GetHardwareInfoVDS method >2014-04-29 14:10:00,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:10:00,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@17324fb2 >2014-04-29 14:10:00,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] HostName = 10.34.27.209 >2014-04-29 14:10:00,851 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:10:00,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3dac8d40] FINISH, GetHardwareInfoVDSCommand, log id: 729378b6 >2014-04-29 14:10:00,856 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [3dac8d40] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:10:00,867 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [11866d32] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:10:00,869 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:10:00,870 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: d6a1f79 >2014-04-29 14:10:00,884 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:10:00,885 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:10:01,969 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 351912c9 >2014-04-29 14:10:02,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] Failed in GetHardwareInfoVDS method >2014-04-29 14:10:02,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:10:02,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@120655b >2014-04-29 14:10:02,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] HostName = 10.34.27.209 >2014-04-29 14:10:02,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:10:02,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [6e3785ac] FINISH, GetHardwareInfoVDSCommand, log id: 351912c9 >2014-04-29 14:10:02,194 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [6e3785ac] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:10:02,203 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [6e3785ac] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:10:02,415 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-22) [20608754] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:10:02,417 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [7aacb51e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:10:02,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [7aacb51e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4a8b3d0 >2014-04-29 14:13:02,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [7aacb51e] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:13:02,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [7aacb51e] FINISH, ConnectStorageServerVDSCommand, log id: 4a8b3d0 >2014-04-29 14:13:02,431 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [7aacb51e] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:13:02,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3d4ea93e >2014-04-29 14:13:02,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3d4ea93e >2014-04-29 14:13:02,855 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:13:02,865 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-22) [fcb07eb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:13:02,866 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [fcb07eb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7c388dfa >2014-04-29 14:13:02,869 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [fcb07eb] FINISH, SetVdsStatusVDSCommand, log id: 7c388dfa >2014-04-29 14:13:02,875 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [fcb07eb] Correlation ID: fcb07eb, Job ID: 9f8ce7a1-d4a3-4e38-b7c4-b0ad75d245c1, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:13:02,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [fcb07eb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:13:02,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [fcb07eb] Correlation ID: 20608754, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:13:02,887 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-22) [40c946b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:13:02,888 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-22) [18da1d2c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:13:02,889 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [18da1d2c] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-9) [624b825c] Autorecovering 1 hosts >2014-04-29 14:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-9) [624b825c] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 69c6df85 >2014-04-29 14:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] FINISH, SetVdsStatusVDSCommand, log id: 69c6df85 >2014-04-29 14:15:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5361bd2b >2014-04-29 14:15:00,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5eea050e >2014-04-29 14:15:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] Failed in GetHardwareInfoVDS method >2014-04-29 14:15:00,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:15:00,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@55f20ad0 >2014-04-29 14:15:00,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] HostName = 10.34.27.209 >2014-04-29 14:15:00,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:15:00,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [547fea77] FINISH, GetHardwareInfoVDSCommand, log id: 5eea050e >2014-04-29 14:15:00,879 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [547fea77] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:15:00,917 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [1d10a6d8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:15:00,919 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:15:00,920 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5361bd2b >2014-04-29 14:15:00,943 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:15:00,945 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:15:03,233 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 16316b44 >2014-04-29 14:15:03,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) Failed in GetHardwareInfoVDS method >2014-04-29 14:15:03,455 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:15:03,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@54918119 >2014-04-29 14:15:03,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) HostName = 10.34.27.209 >2014-04-29 14:15:03,456 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:15:03,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) FINISH, GetHardwareInfoVDSCommand, log id: 16316b44 >2014-04-29 14:15:03,460 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:15:03,469 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:15:03,685 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-12) [588d3c0c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:15:03,687 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [6dfc947] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:15:03,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [6dfc947] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 95d8270 >2014-04-29 14:18:03,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [6dfc947] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:18:03,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [6dfc947] FINISH, ConnectStorageServerVDSCommand, log id: 95d8270 >2014-04-29 14:18:03,695 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [6dfc947] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:18:03,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: b664ef0 >2014-04-29 14:18:04,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: b664ef0 >2014-04-29 14:18:04,113 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:18:04,125 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [eeb2680] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:18:04,127 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [eeb2680] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 24701934 >2014-04-29 14:18:04,129 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [eeb2680] FINISH, SetVdsStatusVDSCommand, log id: 24701934 >2014-04-29 14:18:04,140 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [eeb2680] Correlation ID: eeb2680, Job ID: fc14592c-07ac-4f10-ac49-fa3d37f353ba, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:18:04,156 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [eeb2680] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:18:04,158 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [eeb2680] Correlation ID: 588d3c0c, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:18:04,164 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [263242fc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:18:04,166 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [25139a66] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:18:04,166 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [25139a66] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-99) [147b6be8] Autorecovering 1 hosts >2014-04-29 14:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-99) [147b6be8] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 16fc4275 >2014-04-29 14:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] FINISH, SetVdsStatusVDSCommand, log id: 16fc4275 >2014-04-29 14:20:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6659caf4 >2014-04-29 14:20:00,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3e4a67dd >2014-04-29 14:20:00,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] Failed in GetHardwareInfoVDS method >2014-04-29 14:20:00,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:20:00,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6e8fe1e >2014-04-29 14:20:00,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] HostName = 10.34.27.209 >2014-04-29 14:20:00,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:20:00,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [723d9852] FINISH, GetHardwareInfoVDSCommand, log id: 3e4a67dd >2014-04-29 14:20:00,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [723d9852] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:20:00,873 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-99) [6fc53006] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:20:00,874 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-99) [181b9a34] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:20:00,875 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) [181b9a34] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6659caf4 >2014-04-29 14:20:00,901 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [181b9a34] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:20:00,904 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [181b9a34] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:20:01,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [65605de0] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1608c12d >2014-04-29 14:20:01,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [65605de0] Failed in GetHardwareInfoVDS method >2014-04-29 14:20:01,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [65605de0] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:20:01,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [65605de0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@e40754c >2014-04-29 14:20:01,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [65605de0] HostName = 10.34.27.209 >2014-04-29 14:20:01,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [65605de0] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:20:01,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [65605de0] FINISH, GetHardwareInfoVDSCommand, log id: 1608c12d >2014-04-29 14:20:01,722 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [65605de0] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:20:01,734 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [65605de0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:20:01,946 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-31) [3dddfddc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:20:01,947 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [7653513c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:20:01,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [7653513c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 415d734c >2014-04-29 14:23:01,955 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [7653513c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:23:01,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [7653513c] FINISH, ConnectStorageServerVDSCommand, log id: 415d734c >2014-04-29 14:23:01,956 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [7653513c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:23:01,961 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7c570aa9 >2014-04-29 14:23:02,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7c570aa9 >2014-04-29 14:23:02,376 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:23:02,387 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-31) [74bfb75] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:23:02,388 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [74bfb75] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 794f771c >2014-04-29 14:23:02,391 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [74bfb75] FINISH, SetVdsStatusVDSCommand, log id: 794f771c >2014-04-29 14:23:02,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [74bfb75] Correlation ID: 74bfb75, Job ID: 7f19df3e-d768-4a62-b31c-e29de5982f0f, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:23:02,403 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [74bfb75] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:23:02,405 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [74bfb75] Correlation ID: 3dddfddc, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:23:02,408 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-31) [401305b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:23:02,409 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-31) [7a93dbee] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:23:02,410 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [7a93dbee] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:25:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) [8088d97] Autorecovering 1 hosts >2014-04-29 14:25:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) [8088d97] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:25:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:25:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:25:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:25:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:25:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4a000e89 >2014-04-29 14:25:00,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] FINISH, SetVdsStatusVDSCommand, log id: 4a000e89 >2014-04-29 14:25:00,020 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 62d6348 >2014-04-29 14:25:00,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4203c404 >2014-04-29 14:25:00,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] Failed in GetHardwareInfoVDS method >2014-04-29 14:25:00,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:25:00,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3122fc88 >2014-04-29 14:25:00,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] HostName = 10.34.27.209 >2014-04-29 14:25:00,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:25:00,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [dc6e027] FINISH, GetHardwareInfoVDSCommand, log id: 4203c404 >2014-04-29 14:25:00,850 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [dc6e027] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:25:00,859 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-51) [744b95c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:25:00,860 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-51) [535824f4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:25:00,861 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) [535824f4] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 62d6348 >2014-04-29 14:25:00,873 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [535824f4] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:25:00,874 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [535824f4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:25:02,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1e05f782] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 37601032 >2014-04-29 14:25:02,973 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1e05f782] Failed in GetHardwareInfoVDS method >2014-04-29 14:25:02,973 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1e05f782] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:25:02,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1e05f782] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3c75f05 >2014-04-29 14:25:02,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1e05f782] HostName = 10.34.27.209 >2014-04-29 14:25:02,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1e05f782] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:25:02,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [1e05f782] FINISH, GetHardwareInfoVDSCommand, log id: 37601032 >2014-04-29 14:25:02,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [1e05f782] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:25:03,057 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [1e05f782] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:25:03,266 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-50) [4f99e81b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:25:03,268 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [36e07c81] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:25:03,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [36e07c81] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70667c04 >2014-04-29 14:28:03,277 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [36e07c81] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:28:03,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [36e07c81] FINISH, ConnectStorageServerVDSCommand, log id: 70667c04 >2014-04-29 14:28:03,278 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [36e07c81] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:28:03,282 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 18cf4e4d >2014-04-29 14:28:03,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 18cf4e4d >2014-04-29 14:28:03,711 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:28:03,727 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-50) [52b5f2e6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:28:03,729 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [52b5f2e6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 58f82634 >2014-04-29 14:28:03,731 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [52b5f2e6] FINISH, SetVdsStatusVDSCommand, log id: 58f82634 >2014-04-29 14:28:03,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [52b5f2e6] Correlation ID: 52b5f2e6, Job ID: 3eebd856-045d-4141-ac18-9355fc6e3702, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:28:03,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [52b5f2e6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:28:03,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [52b5f2e6] Correlation ID: 4f99e81b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:28:03,753 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [378f81c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:28:03,754 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [53652354] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:28:03,755 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [53652354] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [2b34b0c9] Autorecovering 1 hosts >2014-04-29 14:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [2b34b0c9] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7a946026 >2014-04-29 14:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] FINISH, SetVdsStatusVDSCommand, log id: 7a946026 >2014-04-29 14:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 650c870b >2014-04-29 14:30:00,616 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 49f0aa64 >2014-04-29 14:30:00,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] Failed in GetHardwareInfoVDS method >2014-04-29 14:30:00,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:30:00,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@27070c35 >2014-04-29 14:30:00,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] HostName = 10.34.27.209 >2014-04-29 14:30:00,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:30:00,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [35cb202e] FINISH, GetHardwareInfoVDSCommand, log id: 49f0aa64 >2014-04-29 14:30:00,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [35cb202e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:30:00,844 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [30c0d18] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:30:00,845 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [43f50b33] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:30:00,846 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [43f50b33] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 650c870b >2014-04-29 14:30:00,861 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [43f50b33] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:30:00,861 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [43f50b33] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:30:04,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [4fe573d7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6964782d >2014-04-29 14:30:04,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Failed in GetHardwareInfoVDS method >2014-04-29 14:30:04,350 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:30:04,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@430e0869 >2014-04-29 14:30:04,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [4fe573d7] HostName = 10.34.27.209 >2014-04-29 14:30:04,353 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:30:04,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [4fe573d7] FINISH, GetHardwareInfoVDSCommand, log id: 6964782d >2014-04-29 14:30:04,358 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:30:04,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [4fe573d7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:30:04,580 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-37) [4761cb5e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:30:04,582 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-37) [29836a53] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:30:04,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [29836a53] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6329a595 >2014-04-29 14:33:04,590 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [29836a53] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:33:04,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [29836a53] FINISH, ConnectStorageServerVDSCommand, log id: 6329a595 >2014-04-29 14:33:04,590 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-37) [29836a53] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:33:04,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4b9b53a9 >2014-04-29 14:33:05,015 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 4b9b53a9 >2014-04-29 14:33:05,015 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:33:05,025 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-37) [f51e4f5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:33:05,026 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [f51e4f5] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 18333faf >2014-04-29 14:33:05,028 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [f51e4f5] FINISH, SetVdsStatusVDSCommand, log id: 18333faf >2014-04-29 14:33:05,034 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [f51e4f5] Correlation ID: f51e4f5, Job ID: 3cad978b-68e9-4453-b05d-9be48f253c2d, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:33:05,041 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [f51e4f5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:33:05,043 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [f51e4f5] Correlation ID: 4761cb5e, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:33:05,047 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-37) [46445b91] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:33:05,048 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-37) [73412e9e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:33:05,049 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [73412e9e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-68) [310e0777] Autorecovering 1 hosts >2014-04-29 14:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-68) [310e0777] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3b11c7b0 >2014-04-29 14:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] FINISH, SetVdsStatusVDSCommand, log id: 3b11c7b0 >2014-04-29 14:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7f6e0859 >2014-04-29 14:35:00,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6fce6add >2014-04-29 14:35:00,837 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] Failed in GetHardwareInfoVDS method >2014-04-29 14:35:00,837 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:35:00,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@471de169 >2014-04-29 14:35:00,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] HostName = 10.34.27.209 >2014-04-29 14:35:00,838 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:35:00,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [62f29087] FINISH, GetHardwareInfoVDSCommand, log id: 6fce6add >2014-04-29 14:35:00,843 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [62f29087] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:35:00,850 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [4ffcd474] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:35:00,851 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:35:00,852 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7f6e0859 >2014-04-29 14:35:00,867 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:35:00,868 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:35:02,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 31f8c33e >2014-04-29 14:35:02,602 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] Failed in GetHardwareInfoVDS method >2014-04-29 14:35:02,602 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:35:02,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@42da3513 >2014-04-29 14:35:02,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] HostName = 10.34.27.209 >2014-04-29 14:35:02,602 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:35:02,602 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [13b27885] FINISH, GetHardwareInfoVDSCommand, log id: 31f8c33e >2014-04-29 14:35:02,614 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [13b27885] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:35:02,638 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [13b27885] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:35:02,848 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-69) [1e4eb721] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:35:02,849 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [1710d353] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:35:02,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [1710d353] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3f180ec3 >2014-04-29 14:36:04,991 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Start refreshing all users data >2014-04-29 14:36:07,245 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) [639730bf] Attempting to update VMs/Templates Ovf. >2014-04-29 14:36:07,246 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) [639730bf] Attempting to update VM OVFs in Data Center NFS >2014-04-29 14:36:07,247 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) [639730bf] Successfully updated VM OVFs in Data Center NFS >2014-04-29 14:36:07,247 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) [639730bf] Attempting to update template OVFs in Data Center NFS >2014-04-29 14:36:07,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) [639730bf] Successfully updated templates OVFs in Data Center NFS >2014-04-29 14:36:07,248 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) [639730bf] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 14:36:07,249 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-73) [639730bf] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 14:38:02,859 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [1710d353] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:38:02,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [1710d353] FINISH, ConnectStorageServerVDSCommand, log id: 3f180ec3 >2014-04-29 14:38:02,859 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [1710d353] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:38:02,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 57fc7fc1 >2014-04-29 14:38:03,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 57fc7fc1 >2014-04-29 14:38:03,302 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:38:03,313 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-69) [145d27ae] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:38:03,314 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [145d27ae] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5d2fdc90 >2014-04-29 14:38:03,317 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [145d27ae] FINISH, SetVdsStatusVDSCommand, log id: 5d2fdc90 >2014-04-29 14:38:03,323 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [145d27ae] Correlation ID: 145d27ae, Job ID: 70565050-1b6a-4034-ab90-bdf3c714ac76, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:38:03,330 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [145d27ae] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:38:03,332 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [145d27ae] Correlation ID: 1e4eb721, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:38:03,336 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-69) [740601a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:38:03,337 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-69) [420ef9d5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:38:03,338 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [420ef9d5] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-57) Autorecovering 1 hosts >2014-04-29 14:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-57) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5fc8ec9c >2014-04-29 14:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] FINISH, SetVdsStatusVDSCommand, log id: 5fc8ec9c >2014-04-29 14:40:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6499238c >2014-04-29 14:40:00,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 28a49cfb >2014-04-29 14:40:00,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] Failed in GetHardwareInfoVDS method >2014-04-29 14:40:00,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:40:00,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4f3e3795 >2014-04-29 14:40:00,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] HostName = 10.34.27.209 >2014-04-29 14:40:00,826 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:40:00,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [22ccc591] FINISH, GetHardwareInfoVDSCommand, log id: 28a49cfb >2014-04-29 14:40:00,831 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [22ccc591] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:40:00,840 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [17180651] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:40:00,842 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [46196251] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:40:00,843 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-57) [46196251] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6499238c >2014-04-29 14:40:00,863 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [46196251] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:40:00,864 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [46196251] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:40:03,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [c79bcbe] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4a5cf210 >2014-04-29 14:40:03,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [c79bcbe] Failed in GetHardwareInfoVDS method >2014-04-29 14:40:03,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [c79bcbe] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:40:03,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [c79bcbe] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1c181e94 >2014-04-29 14:40:03,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [c79bcbe] HostName = 10.34.27.209 >2014-04-29 14:40:03,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [c79bcbe] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:40:03,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [c79bcbe] FINISH, GetHardwareInfoVDSCommand, log id: 4a5cf210 >2014-04-29 14:40:03,902 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [c79bcbe] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:40:03,911 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [c79bcbe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:40:04,119 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-89) [5eba1700] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:40:04,121 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [1d3a3e0f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:40:04,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [1d3a3e0f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 96a7fd0 >2014-04-29 14:43:04,133 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [1d3a3e0f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:43:04,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [1d3a3e0f] FINISH, ConnectStorageServerVDSCommand, log id: 96a7fd0 >2014-04-29 14:43:04,133 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [1d3a3e0f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:43:04,136 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5ec6c170 >2014-04-29 14:43:04,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5ec6c170 >2014-04-29 14:43:04,565 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:43:04,582 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-89) [7124285d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:43:04,583 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [7124285d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6afdfcfe >2014-04-29 14:43:04,586 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [7124285d] FINISH, SetVdsStatusVDSCommand, log id: 6afdfcfe >2014-04-29 14:43:04,592 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [7124285d] Correlation ID: 7124285d, Job ID: 3c50e390-4034-4371-9f55-b0aa85b64f0f, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:43:04,598 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [7124285d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:43:04,600 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [7124285d] Correlation ID: 5eba1700, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:43:04,606 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [5d4bced6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:43:04,608 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [403a46e5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:43:04,609 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [403a46e5] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-99) [181b9a34] Autorecovering 1 hosts >2014-04-29 14:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-99) [181b9a34] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3125ea80 >2014-04-29 14:45:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] FINISH, SetVdsStatusVDSCommand, log id: 3125ea80 >2014-04-29 14:45:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4801c2be >2014-04-29 14:45:00,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 732c35ae >2014-04-29 14:45:00,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] Failed in GetHardwareInfoVDS method >2014-04-29 14:45:00,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:45:00,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@38bf29c >2014-04-29 14:45:00,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] HostName = 10.34.27.209 >2014-04-29 14:45:00,766 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:45:00,766 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [603b58dc] FINISH, GetHardwareInfoVDSCommand, log id: 732c35ae >2014-04-29 14:45:00,778 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [603b58dc] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:45:00,787 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-99) [362fc201] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:45:00,788 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:45:00,789 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4801c2be >2014-04-29 14:45:00,803 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:45:00,804 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:45:01,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7a23254] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1eb4113 >2014-04-29 14:45:02,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7a23254] Failed in GetHardwareInfoVDS method >2014-04-29 14:45:02,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7a23254] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:45:02,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7a23254] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7bfd4760 >2014-04-29 14:45:02,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7a23254] HostName = 10.34.27.209 >2014-04-29 14:45:02,162 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7a23254] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:45:02,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7a23254] FINISH, GetHardwareInfoVDSCommand, log id: 1eb4113 >2014-04-29 14:45:02,166 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [7a23254] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:45:02,175 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [7a23254] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:45:02,387 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-78) [17f54af9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:45:02,388 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [148ae9ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:45:02,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [148ae9ed] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7377886f >2014-04-29 14:48:02,397 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [148ae9ed] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:48:02,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [148ae9ed] FINISH, ConnectStorageServerVDSCommand, log id: 7377886f >2014-04-29 14:48:02,397 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [148ae9ed] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:48:02,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5240f193 >2014-04-29 14:48:02,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5240f193 >2014-04-29 14:48:02,821 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:48:02,833 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-78) [4aa34b3b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:48:02,834 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [4aa34b3b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 75b3b17e >2014-04-29 14:48:02,837 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [4aa34b3b] FINISH, SetVdsStatusVDSCommand, log id: 75b3b17e >2014-04-29 14:48:02,843 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4aa34b3b] Correlation ID: 4aa34b3b, Job ID: bc1fddfb-3a14-420d-bab2-5b619e1ab0a3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:48:02,853 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4aa34b3b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:48:02,855 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4aa34b3b] Correlation ID: 17f54af9, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:48:02,873 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [24b6ac40] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:48:02,875 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [7021f659] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:48:02,875 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [7021f659] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-42) Autorecovering 1 hosts >2014-04-29 14:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-42) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7fb40c12 >2014-04-29 14:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] FINISH, SetVdsStatusVDSCommand, log id: 7fb40c12 >2014-04-29 14:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4a3cef9d >2014-04-29 14:50:00,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 369ebec >2014-04-29 14:50:00,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] Failed in GetHardwareInfoVDS method >2014-04-29 14:50:00,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:50:00,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1effd4b4 >2014-04-29 14:50:00,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] HostName = 10.34.27.209 >2014-04-29 14:50:00,947 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:50:00,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [22ab31cd] FINISH, GetHardwareInfoVDSCommand, log id: 369ebec >2014-04-29 14:50:00,952 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [22ab31cd] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:50:00,960 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-42) [3d34ace6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:50:00,961 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-42) [796f9661] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:50:00,962 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) [796f9661] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4a3cef9d >2014-04-29 14:50:00,976 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [796f9661] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:50:00,977 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [796f9661] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:50:03,220 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [4a646c36] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6b9cff3b >2014-04-29 14:50:03,432 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [4a646c36] Failed in GetHardwareInfoVDS method >2014-04-29 14:50:03,432 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [4a646c36] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:50:03,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [4a646c36] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1038b1af >2014-04-29 14:50:03,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [4a646c36] HostName = 10.34.27.209 >2014-04-29 14:50:03,432 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [4a646c36] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:50:03,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [4a646c36] FINISH, GetHardwareInfoVDSCommand, log id: 6b9cff3b >2014-04-29 14:50:03,456 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [4a646c36] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:50:03,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [4a646c36] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:50:03,680 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-66) [7645dfe5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:50:03,681 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [502f586e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:50:03,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [502f586e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6f0a14a7 >2014-04-29 14:53:03,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [502f586e] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:53:03,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [502f586e] FINISH, ConnectStorageServerVDSCommand, log id: 6f0a14a7 >2014-04-29 14:53:03,690 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [502f586e] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:53:03,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 26de80c9 >2014-04-29 14:53:04,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 26de80c9 >2014-04-29 14:53:04,114 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:53:04,127 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-66) [787b59a8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:53:04,128 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [787b59a8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 545a07a3 >2014-04-29 14:53:04,131 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [787b59a8] FINISH, SetVdsStatusVDSCommand, log id: 545a07a3 >2014-04-29 14:53:04,138 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [787b59a8] Correlation ID: 787b59a8, Job ID: f8f6c108-eabf-45ae-bf7b-34fe0f3e7655, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:53:04,144 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [787b59a8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:53:04,147 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [787b59a8] Correlation ID: 7645dfe5, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:53:04,151 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-66) [347602a1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:53:04,152 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-66) [1160a777] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:53:04,153 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [1160a777] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 14:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-86) [7027322b] Autorecovering 1 hosts >2014-04-29 14:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-86) [7027322b] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 14:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 14:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 14:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7fe8d5ff >2014-04-29 14:55:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] FINISH, SetVdsStatusVDSCommand, log id: 7fe8d5ff >2014-04-29 14:55:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 344ab745 >2014-04-29 14:55:00,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7b9e4d25 >2014-04-29 14:55:00,753 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] Failed in GetHardwareInfoVDS method >2014-04-29 14:55:00,753 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:55:00,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@574d067e >2014-04-29 14:55:00,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] HostName = 10.34.27.209 >2014-04-29 14:55:00,753 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:55:00,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [767f7f82] FINISH, GetHardwareInfoVDSCommand, log id: 7b9e4d25 >2014-04-29 14:55:00,762 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [767f7f82] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:55:00,772 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [36407c5c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:55:00,773 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:55:00,773 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 344ab745 >2014-04-29 14:55:00,787 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 14:55:00,788 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 14:55:01,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1d8e56ad >2014-04-29 14:55:01,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) Failed in GetHardwareInfoVDS method >2014-04-29 14:55:01,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:55:01,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6bdc8703 >2014-04-29 14:55:01,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) HostName = 10.34.27.209 >2014-04-29 14:55:01,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 14:55:01,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) FINISH, GetHardwareInfoVDSCommand, log id: 1d8e56ad >2014-04-29 14:55:01,722 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 14:55:01,730 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 14:55:01,943 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-67) [6b062367] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:55:01,945 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-67) [54dcf03d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 14:55:01,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-67) [54dcf03d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: b9b5ba0 >2014-04-29 14:58:01,952 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-67) [54dcf03d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 14:58:01,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-67) [54dcf03d] FINISH, ConnectStorageServerVDSCommand, log id: b9b5ba0 >2014-04-29 14:58:01,952 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-67) [54dcf03d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 14:58:01,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7a2c7865 >2014-04-29 14:58:02,377 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7a2c7865 >2014-04-29 14:58:02,378 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 14:58:02,389 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-67) [5a62719e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:58:02,390 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-67) [5a62719e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 548d8adf >2014-04-29 14:58:02,392 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-67) [5a62719e] FINISH, SetVdsStatusVDSCommand, log id: 548d8adf >2014-04-29 14:58:02,399 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [5a62719e] Correlation ID: 5a62719e, Job ID: e86d90b4-e51b-4e64-ac6c-668b0e8cc461, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 14:58:02,407 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [5a62719e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 14:58:02,410 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [5a62719e] Correlation ID: 6b062367, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 14:58:02,414 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-67) [b4f6352] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:58:02,415 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-67) [65d81bca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 14:58:02,416 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-67) [65d81bca] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-12) [25139a66] Autorecovering 1 hosts >2014-04-29 15:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-12) [25139a66] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5893dd95 >2014-04-29 15:00:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] FINISH, SetVdsStatusVDSCommand, log id: 5893dd95 >2014-04-29 15:00:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7d2ef23f >2014-04-29 15:00:00,452 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 66e6e84f >2014-04-29 15:00:00,820 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] Failed in GetHardwareInfoVDS method >2014-04-29 15:00:00,821 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:00:00,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@419dcf50 >2014-04-29 15:00:00,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] HostName = 10.34.27.209 >2014-04-29 15:00:00,821 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:00:00,821 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [605db59e] FINISH, GetHardwareInfoVDSCommand, log id: 66e6e84f >2014-04-29 15:00:00,827 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [605db59e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:00:00,837 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [5432c827] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:00:00,838 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:00:00,839 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7d2ef23f >2014-04-29 15:00:00,853 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:00:00,854 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:00:02,785 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 36533016 >2014-04-29 15:00:02,996 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] Failed in GetHardwareInfoVDS method >2014-04-29 15:00:02,996 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:00:02,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@f7a1037 >2014-04-29 15:00:02,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] HostName = 10.34.27.209 >2014-04-29 15:00:02,997 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:00:02,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [5b602e22] FINISH, GetHardwareInfoVDSCommand, log id: 36533016 >2014-04-29 15:00:03,002 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [5b602e22] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:00:03,011 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [5b602e22] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:00:03,232 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-21) [3cffa372] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:00:03,237 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-21) [2473c5e8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:00:03,243 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [2473c5e8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 89c7e55 >2014-04-29 15:03:03,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [2473c5e8] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:03:03,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [2473c5e8] FINISH, ConnectStorageServerVDSCommand, log id: 89c7e55 >2014-04-29 15:03:03,245 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-21) [2473c5e8] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:03:03,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1db39535 >2014-04-29 15:03:03,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1db39535 >2014-04-29 15:03:03,665 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:03:03,677 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-21) [462a0cdc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:03:03,678 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [462a0cdc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 70bf6591 >2014-04-29 15:03:03,681 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [462a0cdc] FINISH, SetVdsStatusVDSCommand, log id: 70bf6591 >2014-04-29 15:03:03,691 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [462a0cdc] Correlation ID: 462a0cdc, Job ID: f73668b4-6598-40a7-b9f2-468f4f1944ce, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:03:03,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [462a0cdc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:03:03,700 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [462a0cdc] Correlation ID: 3cffa372, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:03:03,704 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-21) [5b51a1b5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:03:03,705 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-21) [4fd42b14] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:03:03,705 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) [4fd42b14] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-40) [5c5e48bc] Autorecovering 1 hosts >2014-04-29 15:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-40) [5c5e48bc] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2b38ca74 >2014-04-29 15:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] FINISH, SetVdsStatusVDSCommand, log id: 2b38ca74 >2014-04-29 15:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 29f4e353 >2014-04-29 15:05:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 40de0aee >2014-04-29 15:05:00,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] Failed in GetHardwareInfoVDS method >2014-04-29 15:05:00,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:05:00,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6afe8884 >2014-04-29 15:05:00,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] HostName = 10.34.27.209 >2014-04-29 15:05:00,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:05:00,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [29ae9db5] FINISH, GetHardwareInfoVDSCommand, log id: 40de0aee >2014-04-29 15:05:00,980 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [29ae9db5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:05:00,989 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-40) [57057d6d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:05:00,990 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-40) [10c242c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:05:00,991 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) [10c242c0] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 29f4e353 >2014-04-29 15:05:01,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [10c242c0] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:05:01,011 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [10c242c0] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:05:04,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [797cbadd] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1a74936e >2014-04-29 15:05:04,396 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [797cbadd] Failed in GetHardwareInfoVDS method >2014-04-29 15:05:04,396 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [797cbadd] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:05:04,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [797cbadd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5b7b2853 >2014-04-29 15:05:04,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [797cbadd] HostName = 10.34.27.209 >2014-04-29 15:05:04,396 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [797cbadd] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:05:04,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [797cbadd] FINISH, GetHardwareInfoVDSCommand, log id: 1a74936e >2014-04-29 15:05:04,401 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [797cbadd] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:05:04,410 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [797cbadd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:05:04,620 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [2e4aa1e9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:05:04,621 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [16825cc2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:05:04,634 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [16825cc2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7dfaf85e >2014-04-29 15:08:04,636 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [16825cc2] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:08:04,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [16825cc2] FINISH, ConnectStorageServerVDSCommand, log id: 7dfaf85e >2014-04-29 15:08:04,636 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [16825cc2] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:08:04,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 51aad6e1 >2014-04-29 15:08:05,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 51aad6e1 >2014-04-29 15:08:05,059 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:08:05,071 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-20) [2a2330f4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:08:05,072 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [2a2330f4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 409a82e2 >2014-04-29 15:08:05,074 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [2a2330f4] FINISH, SetVdsStatusVDSCommand, log id: 409a82e2 >2014-04-29 15:08:05,080 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [2a2330f4] Correlation ID: 2a2330f4, Job ID: b44b3e1e-ec1e-4069-bae0-8700c94e334c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:08:05,087 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [2a2330f4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:08:05,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [2a2330f4] Correlation ID: 2e4aa1e9, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:08:05,094 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [2d05292a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:08:05,095 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [541ebd1e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:08:05,095 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) [541ebd1e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-36) [2a1d265f] Autorecovering 1 hosts >2014-04-29 15:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-36) [2a1d265f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:10:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2fe256fd >2014-04-29 15:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] FINISH, SetVdsStatusVDSCommand, log id: 2fe256fd >2014-04-29 15:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7245e693 >2014-04-29 15:10:00,466 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 79dd1dca >2014-04-29 15:10:00,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] Failed in GetHardwareInfoVDS method >2014-04-29 15:10:00,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:10:00,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@23cef747 >2014-04-29 15:10:00,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] HostName = 10.34.27.209 >2014-04-29 15:10:00,836 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:10:00,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [5a46769] FINISH, GetHardwareInfoVDSCommand, log id: 79dd1dca >2014-04-29 15:10:00,841 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [5a46769] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:10:00,850 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [6447a8f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:10:00,851 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [5fb8b447] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:10:00,852 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-36) [5fb8b447] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7245e693 >2014-04-29 15:10:00,866 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [5fb8b447] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:10:00,869 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [5fb8b447] Correlation ID: 5a46769, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was autorecovered. >2014-04-29 15:10:00,869 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [5fb8b447] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:10:02,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [53652354] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4ada4a4e >2014-04-29 15:10:02,650 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [53652354] Failed in GetHardwareInfoVDS method >2014-04-29 15:10:02,650 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [53652354] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:10:02,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [53652354] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@65c4cdf2 >2014-04-29 15:10:02,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [53652354] HostName = 10.34.27.209 >2014-04-29 15:10:02,650 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [53652354] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:10:02,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [53652354] FINISH, GetHardwareInfoVDSCommand, log id: 4ada4a4e >2014-04-29 15:10:02,654 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [53652354] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:10:02,664 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [53652354] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:10:02,873 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-50) [3bd0bd03] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:10:02,875 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [6d6864c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:10:02,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [6d6864c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4bb3c8cd >2014-04-29 15:13:02,885 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [6d6864c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:13:02,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [6d6864c] FINISH, ConnectStorageServerVDSCommand, log id: 4bb3c8cd >2014-04-29 15:13:02,885 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [6d6864c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:13:02,890 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 17366ff3 >2014-04-29 15:13:03,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 17366ff3 >2014-04-29 15:13:03,321 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:13:03,331 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-50) [1ba241c7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:13:03,332 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [1ba241c7] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 313104fc >2014-04-29 15:13:03,335 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [1ba241c7] FINISH, SetVdsStatusVDSCommand, log id: 313104fc >2014-04-29 15:13:03,341 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [1ba241c7] Correlation ID: 1ba241c7, Job ID: d568a90d-ce15-4798-af90-21e0078830f4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:13:03,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [1ba241c7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:13:03,348 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [1ba241c7] Correlation ID: 3bd0bd03, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:13:03,351 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [696a3302] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:13:03,353 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [7a93afa1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:13:03,353 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [7a93afa1] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-74) [64ca0b80] Autorecovering 1 hosts >2014-04-29 15:15:00,001 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-74) [64ca0b80] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:15:00,002 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:15:00,002 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:15:00,002 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:15:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:15:00,006 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2da8abed >2014-04-29 15:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] FINISH, SetVdsStatusVDSCommand, log id: 2da8abed >2014-04-29 15:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 57ecbbc4 >2014-04-29 15:15:00,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 27d11fc3 >2014-04-29 15:15:00,828 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] Failed in GetHardwareInfoVDS method >2014-04-29 15:15:00,829 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:15:00,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6e4938f8 >2014-04-29 15:15:00,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] HostName = 10.34.27.209 >2014-04-29 15:15:00,829 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:15:00,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [6127eee] FINISH, GetHardwareInfoVDSCommand, log id: 27d11fc3 >2014-04-29 15:15:00,834 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [6127eee] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:15:00,842 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-74) [6f39fd51] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:15:00,843 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:15:00,844 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 57ecbbc4 >2014-04-29 15:15:00,859 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:15:00,860 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:15:03,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 42d19dae >2014-04-29 15:15:03,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] Failed in GetHardwareInfoVDS method >2014-04-29 15:15:03,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:15:03,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4fc72cc7 >2014-04-29 15:15:03,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] HostName = 10.34.27.209 >2014-04-29 15:15:03,918 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:15:03,918 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [631f2a4b] FINISH, GetHardwareInfoVDSCommand, log id: 42d19dae >2014-04-29 15:15:03,924 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [631f2a4b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:15:03,936 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [631f2a4b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:15:04,147 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-15) [6dbc608e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:15:04,149 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-15) [506fd428] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:15:04,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [506fd428] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3dc27941 >2014-04-29 15:18:04,157 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [506fd428] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:18:04,157 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [506fd428] FINISH, ConnectStorageServerVDSCommand, log id: 3dc27941 >2014-04-29 15:18:04,157 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-15) [506fd428] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:18:04,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 479dbfff >2014-04-29 15:18:04,589 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 479dbfff >2014-04-29 15:18:04,589 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:18:04,601 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-15) [3a152a87] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:18:04,602 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [3a152a87] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 302692a6 >2014-04-29 15:18:04,606 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [3a152a87] FINISH, SetVdsStatusVDSCommand, log id: 302692a6 >2014-04-29 15:18:04,612 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [3a152a87] Correlation ID: 3a152a87, Job ID: 435e8a2b-aa1d-4c5b-82f5-a53073655c58, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:18:04,618 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [3a152a87] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:18:04,620 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [3a152a87] Correlation ID: 6dbc608e, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:18:04,624 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-15) [1a6ab98c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:18:04,626 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-15) [4a4eba78] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:18:04,628 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [4a4eba78] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [68acb3c9] Autorecovering 1 hosts >2014-04-29 15:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [68acb3c9] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [29569629] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [29569629] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [29569629] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [29569629] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [29569629] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 21010d7f >2014-04-29 15:20:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [29569629] FINISH, SetVdsStatusVDSCommand, log id: 21010d7f >2014-04-29 15:20:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) [29569629] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3ebc5034 >2014-04-29 15:20:00,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [29569629] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2e01a73b >2014-04-29 15:20:00,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [29569629] Failed in GetHardwareInfoVDS method >2014-04-29 15:20:00,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [29569629] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:20:00,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [29569629] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6e8a2a15 >2014-04-29 15:20:00,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [29569629] HostName = 10.34.27.209 >2014-04-29 15:20:00,827 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [29569629] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:20:00,827 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [29569629] FINISH, GetHardwareInfoVDSCommand, log id: 2e01a73b >2014-04-29 15:20:00,832 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [29569629] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:20:00,841 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-65) [495dde7f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:20:00,842 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:20:00,843 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3ebc5034 >2014-04-29 15:20:00,857 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:20:00,858 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:20:01,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [27211418] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1dbe820e >2014-04-29 15:20:02,176 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [27211418] Failed in GetHardwareInfoVDS method >2014-04-29 15:20:02,176 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [27211418] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:20:02,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [27211418] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@464c7463 >2014-04-29 15:20:02,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [27211418] HostName = 10.34.27.209 >2014-04-29 15:20:02,176 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [27211418] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:20:02,176 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [27211418] FINISH, GetHardwareInfoVDSCommand, log id: 1dbe820e >2014-04-29 15:20:02,182 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [27211418] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:20:02,192 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [27211418] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:20:02,403 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-28) [15ace791] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:20:02,405 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-28) [1f2fdc24] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:20:02,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [1f2fdc24] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 10c3bdc >2014-04-29 15:23:02,413 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [1f2fdc24] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:23:02,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [1f2fdc24] FINISH, ConnectStorageServerVDSCommand, log id: 10c3bdc >2014-04-29 15:23:02,413 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-28) [1f2fdc24] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:23:02,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 40fc9c6f >2014-04-29 15:23:02,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 40fc9c6f >2014-04-29 15:23:02,831 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:23:02,842 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-28) [5f10407a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:23:02,843 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [5f10407a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5098a7be >2014-04-29 15:23:02,846 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [5f10407a] FINISH, SetVdsStatusVDSCommand, log id: 5098a7be >2014-04-29 15:23:02,852 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [5f10407a] Correlation ID: 5f10407a, Job ID: 8964b3d6-3968-449a-8fa2-209393a857b0, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:23:02,859 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [5f10407a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:23:02,861 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [5f10407a] Correlation ID: 15ace791, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:23:02,865 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-28) [3336e5d8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:23:02,866 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-28) [2fd2615] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:23:02,867 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [2fd2615] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) Autorecovering 1 hosts >2014-04-29 15:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:25:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 50b61270 >2014-04-29 15:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] FINISH, SetVdsStatusVDSCommand, log id: 50b61270 >2014-04-29 15:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4a3dfcd2 >2014-04-29 15:25:00,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3005439 >2014-04-29 15:25:00,830 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] Failed in GetHardwareInfoVDS method >2014-04-29 15:25:00,830 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:25:00,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@a996bb9 >2014-04-29 15:25:00,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] HostName = 10.34.27.209 >2014-04-29 15:25:00,830 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:25:00,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [61acaa44] FINISH, GetHardwareInfoVDSCommand, log id: 3005439 >2014-04-29 15:25:00,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [61acaa44] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:25:00,844 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-85) [24b1380a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:25:00,845 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:25:00,846 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4a3dfcd2 >2014-04-29 15:25:00,859 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:25:00,860 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:25:03,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3b551772 >2014-04-29 15:25:03,415 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] Failed in GetHardwareInfoVDS method >2014-04-29 15:25:03,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:25:03,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4f765bcc >2014-04-29 15:25:03,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] HostName = 10.34.27.209 >2014-04-29 15:25:03,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:25:03,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [463ee345] FINISH, GetHardwareInfoVDSCommand, log id: 3b551772 >2014-04-29 15:25:03,421 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [463ee345] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:25:03,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [463ee345] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:25:03,647 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-6) [2aa5af14] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:25:03,656 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-6) [484e6f25] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:25:03,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [484e6f25] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 239d9462 >2014-04-29 15:28:03,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [484e6f25] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:28:03,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [484e6f25] FINISH, ConnectStorageServerVDSCommand, log id: 239d9462 >2014-04-29 15:28:03,680 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-6) [484e6f25] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:28:03,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 388845e3 >2014-04-29 15:28:04,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 388845e3 >2014-04-29 15:28:04,108 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:28:04,120 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-6) [69ba1c9d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:28:04,122 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [69ba1c9d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 330a3f6f >2014-04-29 15:28:04,125 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [69ba1c9d] FINISH, SetVdsStatusVDSCommand, log id: 330a3f6f >2014-04-29 15:28:04,131 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [69ba1c9d] Correlation ID: 69ba1c9d, Job ID: 93baa239-af85-44ac-b547-2fb860ed4445, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:28:04,140 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [69ba1c9d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:28:04,142 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [69ba1c9d] Correlation ID: 2aa5af14, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:28:04,146 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [adc18de] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:28:04,147 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [7f3793f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:28:04,148 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [7f3793f3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-10) Autorecovering 1 hosts >2014-04-29 15:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-10) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3be83304 >2014-04-29 15:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] FINISH, SetVdsStatusVDSCommand, log id: 3be83304 >2014-04-29 15:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 68320b26 >2014-04-29 15:30:00,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7f58bafe >2014-04-29 15:30:00,838 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] Failed in GetHardwareInfoVDS method >2014-04-29 15:30:00,838 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:30:00,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1a5cbb03 >2014-04-29 15:30:00,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] HostName = 10.34.27.209 >2014-04-29 15:30:00,838 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:30:00,838 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [24a948a6] FINISH, GetHardwareInfoVDSCommand, log id: 7f58bafe >2014-04-29 15:30:00,843 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [24a948a6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:30:00,853 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-10) [63298bf0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:30:00,854 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:30:00,855 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 68320b26 >2014-04-29 15:30:00,881 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:30:00,882 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:30:01,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 47fcc247 >2014-04-29 15:30:01,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] Failed in GetHardwareInfoVDS method >2014-04-29 15:30:01,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:30:01,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5bc75ead >2014-04-29 15:30:01,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] HostName = 10.34.27.209 >2014-04-29 15:30:01,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:30:01,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [2c79e878] FINISH, GetHardwareInfoVDSCommand, log id: 47fcc247 >2014-04-29 15:30:01,742 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [2c79e878] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:30:01,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [2c79e878] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:30:02,083 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-82) [1d7c2906] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:30:02,092 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-82) [32050abd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:30:02,104 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [32050abd] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: a83f6f9 >2014-04-29 15:33:02,106 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [32050abd] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:33:02,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [32050abd] FINISH, ConnectStorageServerVDSCommand, log id: a83f6f9 >2014-04-29 15:33:02,106 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-82) [32050abd] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:33:02,112 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 34609fd4 >2014-04-29 15:33:02,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 34609fd4 >2014-04-29 15:33:02,531 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:33:02,545 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-82) [3d19ebb9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:33:02,546 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3d19ebb9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6e8b0e17 >2014-04-29 15:33:02,549 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3d19ebb9] FINISH, SetVdsStatusVDSCommand, log id: 6e8b0e17 >2014-04-29 15:33:02,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3d19ebb9] Correlation ID: 3d19ebb9, Job ID: e5e5f740-1153-47ad-99fb-c9b0ba38d5f0, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:33:02,561 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3d19ebb9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:33:02,563 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3d19ebb9] Correlation ID: 1d7c2906, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:33:02,568 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-82) [154b6c07] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:33:02,569 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-82) [82758c6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:33:02,570 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) [82758c6] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) [621d3184] Autorecovering 1 hosts >2014-04-29 15:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) [621d3184] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:35:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 508e06e0 >2014-04-29 15:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] FINISH, SetVdsStatusVDSCommand, log id: 508e06e0 >2014-04-29 15:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 459ed472 >2014-04-29 15:35:00,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 42eb0f21 >2014-04-29 15:35:00,828 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] Failed in GetHardwareInfoVDS method >2014-04-29 15:35:00,828 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:35:00,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6d83fe69 >2014-04-29 15:35:00,828 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] HostName = 10.34.27.209 >2014-04-29 15:35:00,828 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:35:00,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] FINISH, GetHardwareInfoVDSCommand, log id: 42eb0f21 >2014-04-29 15:35:00,834 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [3e5aa4db] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:35:00,843 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-14) [5d74b144] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:35:00,844 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:35:00,844 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 459ed472 >2014-04-29 15:35:00,858 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:35:00,859 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:35:02,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4a0c67fa >2014-04-29 15:35:03,137 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] Failed in GetHardwareInfoVDS method >2014-04-29 15:35:03,138 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:35:03,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@747e7410 >2014-04-29 15:35:03,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] HostName = 10.34.27.209 >2014-04-29 15:35:03,138 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:35:03,138 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] FINISH, GetHardwareInfoVDSCommand, log id: 4a0c67fa >2014-04-29 15:35:03,143 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:35:03,153 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [7eb56ed2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:35:03,375 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-54) [3ebbdf4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:35:03,377 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-54) [38d65d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:35:03,389 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-54) [38d65d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7cb0a62b >2014-04-29 15:36:04,993 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-22) [18da1d2c] Start refreshing all users data >2014-04-29 15:36:07,250 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VMs/Templates Ovf. >2014-04-29 15:36:07,251 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update VM OVFs in Data Center NFS >2014-04-29 15:36:07,252 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully updated VM OVFs in Data Center NFS >2014-04-29 15:36:07,252 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to update template OVFs in Data Center NFS >2014-04-29 15:36:07,254 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully updated templates OVFs in Data Center NFS >2014-04-29 15:36:07,254 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 15:36:07,255 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-70) Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 15:38:03,391 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-54) [38d65d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:38:03,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-54) [38d65d] FINISH, ConnectStorageServerVDSCommand, log id: 7cb0a62b >2014-04-29 15:38:03,391 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-54) [38d65d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:38:03,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: b537833 >2014-04-29 15:38:03,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: b537833 >2014-04-29 15:38:03,819 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:38:03,833 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-54) [5494f3f0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:38:03,835 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [5494f3f0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 263d6167 >2014-04-29 15:38:03,838 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [5494f3f0] FINISH, SetVdsStatusVDSCommand, log id: 263d6167 >2014-04-29 15:38:03,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [5494f3f0] Correlation ID: 5494f3f0, Job ID: 4d77a8d9-5ffb-4861-b98e-5b3acb38366c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:38:03,859 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [5494f3f0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:38:03,861 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [5494f3f0] Correlation ID: 3ebbdf4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:38:03,866 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-54) [1b10fba1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:38:03,868 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-54) [2bd3cf05] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:38:03,868 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-54) [2bd3cf05] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-60) [29a144e6] Autorecovering 1 hosts >2014-04-29 15:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-60) [29a144e6] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5a6399b2 >2014-04-29 15:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] FINISH, SetVdsStatusVDSCommand, log id: 5a6399b2 >2014-04-29 15:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 46e06e14 >2014-04-29 15:40:00,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3b83a915 >2014-04-29 15:40:00,849 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] Failed in GetHardwareInfoVDS method >2014-04-29 15:40:00,849 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:40:00,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2631c4c6 >2014-04-29 15:40:00,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] HostName = 10.34.27.209 >2014-04-29 15:40:00,849 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:40:00,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [5078e969] FINISH, GetHardwareInfoVDSCommand, log id: 3b83a915 >2014-04-29 15:40:00,854 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [5078e969] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:40:00,862 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-60) [64e9f19e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:40:00,863 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:40:00,864 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 46e06e14 >2014-04-29 15:40:00,877 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:40:00,878 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:40:01,215 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4aa4aa99 >2014-04-29 15:40:01,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] Failed in GetHardwareInfoVDS method >2014-04-29 15:40:01,430 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:40:01,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@bc4918 >2014-04-29 15:40:01,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] HostName = 10.34.27.209 >2014-04-29 15:40:01,430 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:40:01,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [283390e7] FINISH, GetHardwareInfoVDSCommand, log id: 4aa4aa99 >2014-04-29 15:40:01,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [283390e7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:40:01,445 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [283390e7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:40:01,661 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-34) [3574179f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:40:01,663 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [31ace373] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:40:01,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [31ace373] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7da8983a >2014-04-29 15:43:01,674 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [31ace373] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:43:01,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [31ace373] FINISH, ConnectStorageServerVDSCommand, log id: 7da8983a >2014-04-29 15:43:01,675 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [31ace373] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:43:01,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 58f316e9 >2014-04-29 15:43:02,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 58f316e9 >2014-04-29 15:43:02,133 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:43:02,146 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-34) [227da4ce] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:43:02,147 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [227da4ce] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7fb1f94d >2014-04-29 15:43:02,149 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [227da4ce] FINISH, SetVdsStatusVDSCommand, log id: 7fb1f94d >2014-04-29 15:43:02,155 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [227da4ce] Correlation ID: 227da4ce, Job ID: 3d1fb23a-702c-4111-98b4-23acfc93eda7, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:43:02,161 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [227da4ce] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:43:02,163 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [227da4ce] Correlation ID: 3574179f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:43:02,167 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [6d522a1d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:43:02,168 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [5419206a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:43:02,169 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [5419206a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-32) Autorecovering 1 hosts >2014-04-29 15:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-32) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 703fab61 >2014-04-29 15:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] FINISH, SetVdsStatusVDSCommand, log id: 703fab61 >2014-04-29 15:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: be316db >2014-04-29 15:45:00,463 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4ac8a1fb >2014-04-29 15:45:00,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] Failed in GetHardwareInfoVDS method >2014-04-29 15:45:00,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:45:00,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@728d8caa >2014-04-29 15:45:00,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] HostName = 10.34.27.209 >2014-04-29 15:45:00,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:45:00,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] FINISH, GetHardwareInfoVDSCommand, log id: 4ac8a1fb >2014-04-29 15:45:00,851 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [79ccc4cb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:45:00,860 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-32) [38a99d0d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:45:00,861 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-32) [555e6086] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:45:00,862 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) [555e6086] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: be316db >2014-04-29 15:45:00,876 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [555e6086] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:45:00,877 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [555e6086] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:45:02,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5a5a5bca >2014-04-29 15:45:02,748 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] Failed in GetHardwareInfoVDS method >2014-04-29 15:45:02,749 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:45:02,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6240591d >2014-04-29 15:45:02,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] HostName = 10.34.27.209 >2014-04-29 15:45:02,749 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:45:02,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [42bc88ba] FINISH, GetHardwareInfoVDSCommand, log id: 5a5a5bca >2014-04-29 15:45:02,753 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [42bc88ba] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:45:02,763 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [42bc88ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:45:02,977 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-76) [27531246] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:45:02,979 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-76) [4371d90c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:45:02,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-76) [4371d90c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6f5cbc87 >2014-04-29 15:48:02,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-76) [4371d90c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:48:02,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-76) [4371d90c] FINISH, ConnectStorageServerVDSCommand, log id: 6f5cbc87 >2014-04-29 15:48:02,986 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-76) [4371d90c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:48:02,990 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 494ea62c >2014-04-29 15:48:03,435 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 494ea62c >2014-04-29 15:48:03,435 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:48:03,448 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-76) [4d6ca6b2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:48:03,449 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [4d6ca6b2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 52259c4d >2014-04-29 15:48:03,452 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [4d6ca6b2] FINISH, SetVdsStatusVDSCommand, log id: 52259c4d >2014-04-29 15:48:03,464 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [4d6ca6b2] Correlation ID: 4d6ca6b2, Job ID: 2a87f194-38c5-4393-b511-4930f6e801e8, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:48:03,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [4d6ca6b2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:48:03,472 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [4d6ca6b2] Correlation ID: 27531246, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:48:03,476 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-76) [631bea1e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:48:03,477 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-76) [1238a5e6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:48:03,478 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [1238a5e6] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [4031c767] Autorecovering 1 hosts >2014-04-29 15:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [4031c767] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 22a58fc8 >2014-04-29 15:50:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] FINISH, SetVdsStatusVDSCommand, log id: 22a58fc8 >2014-04-29 15:50:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 368db01e >2014-04-29 15:50:00,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7a32409e >2014-04-29 15:50:00,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] Failed in GetHardwareInfoVDS method >2014-04-29 15:50:00,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:50:00,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@366c15a0 >2014-04-29 15:50:00,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] HostName = 10.34.27.209 >2014-04-29 15:50:00,846 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:50:00,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [476005ce] FINISH, GetHardwareInfoVDSCommand, log id: 7a32409e >2014-04-29 15:50:00,852 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [476005ce] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:50:00,865 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [5eea2eed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:50:00,867 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [17b82c51] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:50:00,882 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [17b82c51] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 368db01e >2014-04-29 15:50:00,931 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [17b82c51] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:50:00,932 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [17b82c51] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:50:03,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [403a46e5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 49d1095b >2014-04-29 15:50:04,047 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [403a46e5] Failed in GetHardwareInfoVDS method >2014-04-29 15:50:04,047 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [403a46e5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:50:04,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [403a46e5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7958608 >2014-04-29 15:50:04,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [403a46e5] HostName = 10.34.27.209 >2014-04-29 15:50:04,047 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [403a46e5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:50:04,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [403a46e5] FINISH, GetHardwareInfoVDSCommand, log id: 49d1095b >2014-04-29 15:50:04,051 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [403a46e5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:50:04,064 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [403a46e5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:50:04,286 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-89) [3d1d8572] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:50:04,287 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [3b07ddcc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:50:04,294 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [3b07ddcc] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2b1daa0 >2014-04-29 15:53:04,295 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [3b07ddcc] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:53:04,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [3b07ddcc] FINISH, ConnectStorageServerVDSCommand, log id: 2b1daa0 >2014-04-29 15:53:04,295 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [3b07ddcc] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:53:04,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5b49ab23 >2014-04-29 15:53:04,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5b49ab23 >2014-04-29 15:53:04,742 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:53:04,753 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-89) [5b93823] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:53:04,755 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [5b93823] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7d7986bb >2014-04-29 15:53:04,757 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [5b93823] FINISH, SetVdsStatusVDSCommand, log id: 7d7986bb >2014-04-29 15:53:04,763 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [5b93823] Correlation ID: 5b93823, Job ID: 4b0a5dd0-8715-4657-aaf6-8537519f98b6, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:53:04,771 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [5b93823] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:53:04,774 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [5b93823] Correlation ID: 3d1d8572, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:53:04,778 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [52e16689] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:53:04,779 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [58a99b8b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:53:04,780 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [58a99b8b] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 15:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) [7021f659] Autorecovering 1 hosts >2014-04-29 15:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) [7021f659] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 15:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 15:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 15:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: c7f096b >2014-04-29 15:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] FINISH, SetVdsStatusVDSCommand, log id: c7f096b >2014-04-29 15:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: f959f20 >2014-04-29 15:55:00,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6a0b8235 >2014-04-29 15:55:00,842 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] Failed in GetHardwareInfoVDS method >2014-04-29 15:55:00,843 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:55:00,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2e6589f0 >2014-04-29 15:55:00,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] HostName = 10.34.27.209 >2014-04-29 15:55:00,843 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:55:00,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [13acb4fa] FINISH, GetHardwareInfoVDSCommand, log id: 6a0b8235 >2014-04-29 15:55:00,871 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [13acb4fa] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:55:00,922 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [10f5f3c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:55:00,923 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [52e4e6d5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:55:00,924 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [52e4e6d5] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: f959f20 >2014-04-29 15:55:00,938 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [52e4e6d5] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 15:55:00,939 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [52e4e6d5] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 15:55:02,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7a0b6f35 >2014-04-29 15:55:02,347 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] Failed in GetHardwareInfoVDS method >2014-04-29 15:55:02,347 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:55:02,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4b23f2c >2014-04-29 15:55:02,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] HostName = 10.34.27.209 >2014-04-29 15:55:02,348 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 15:55:02,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] FINISH, GetHardwareInfoVDSCommand, log id: 7a0b6f35 >2014-04-29 15:55:02,352 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 15:55:02,362 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [1e7d28c1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 15:55:02,577 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-99) [7330e0e3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:55:02,579 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-99) [9d5f488] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 15:55:02,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-99) [9d5f488] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 357e30b4 >2014-04-29 15:58:02,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-99) [9d5f488] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 15:58:02,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-99) [9d5f488] FINISH, ConnectStorageServerVDSCommand, log id: 357e30b4 >2014-04-29 15:58:02,587 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-99) [9d5f488] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 15:58:02,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 53786d0a >2014-04-29 15:58:03,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 53786d0a >2014-04-29 15:58:03,029 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 15:58:03,041 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-99) [44794423] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:58:03,042 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [44794423] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 46682475 >2014-04-29 15:58:03,045 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-99) [44794423] FINISH, SetVdsStatusVDSCommand, log id: 46682475 >2014-04-29 15:58:03,051 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [44794423] Correlation ID: 44794423, Job ID: 538edda0-a4c5-4f85-af4f-50a0d3be4330, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 15:58:03,057 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [44794423] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 15:58:03,059 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-99) [44794423] Correlation ID: 7330e0e3, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 15:58:03,064 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-99) [2c77ec3d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:58:03,066 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-99) [5845dadb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 15:58:03,066 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-99) [5845dadb] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-16) Autorecovering 1 hosts >2014-04-29 16:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-16) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 59c10f75 >2014-04-29 16:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] FINISH, SetVdsStatusVDSCommand, log id: 59c10f75 >2014-04-29 16:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2127f8f0 >2014-04-29 16:00:00,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4ce17631 >2014-04-29 16:00:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] Failed in GetHardwareInfoVDS method >2014-04-29 16:00:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:00:00,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6616b265 >2014-04-29 16:00:00,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] HostName = 10.34.27.209 >2014-04-29 16:00:00,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:00:00,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [62a35ef7] FINISH, GetHardwareInfoVDSCommand, log id: 4ce17631 >2014-04-29 16:00:00,875 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [62a35ef7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:00:00,883 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-16) [5cc2fff5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:00:00,884 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:00:00,885 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2127f8f0 >2014-04-29 16:00:00,898 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:00:00,899 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:00:03,428 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [1160a777] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 357894de >2014-04-29 16:00:03,652 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [1160a777] Failed in GetHardwareInfoVDS method >2014-04-29 16:00:03,652 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [1160a777] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:00:03,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [1160a777] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7f75963e >2014-04-29 16:00:03,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [1160a777] HostName = 10.34.27.209 >2014-04-29 16:00:03,652 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [1160a777] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:00:03,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-66) [1160a777] FINISH, GetHardwareInfoVDSCommand, log id: 357894de >2014-04-29 16:00:03,657 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [1160a777] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:00:03,666 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [1160a777] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:00:03,892 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-66) [11239f06] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:00:03,894 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [782feb08] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:00:03,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [782feb08] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1f8c5f59 >2014-04-29 16:03:03,902 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [782feb08] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:03:03,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-66) [782feb08] FINISH, ConnectStorageServerVDSCommand, log id: 1f8c5f59 >2014-04-29 16:03:03,902 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-66) [782feb08] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:03:03,906 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1e62beb0 >2014-04-29 16:03:04,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 1e62beb0 >2014-04-29 16:03:04,348 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:03:04,360 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-66) [5d8341cc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:03:04,361 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [5d8341cc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: ddd7210 >2014-04-29 16:03:04,364 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-66) [5d8341cc] FINISH, SetVdsStatusVDSCommand, log id: ddd7210 >2014-04-29 16:03:04,379 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [5d8341cc] Correlation ID: 5d8341cc, Job ID: 4ea195e4-6862-4633-a16a-1f6f6c3d4a85, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:03:04,387 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [5d8341cc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:03:04,389 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-66) [5d8341cc] Correlation ID: 11239f06, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:03:04,393 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-66) [2a6168df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:03:04,394 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-66) [4400bde0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:03:04,394 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-66) [4400bde0] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [3d8c60fc] Autorecovering 1 hosts >2014-04-29 16:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [3d8c60fc] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 67440165 >2014-04-29 16:05:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] FINISH, SetVdsStatusVDSCommand, log id: 67440165 >2014-04-29 16:05:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 57b38b47 >2014-04-29 16:05:00,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5e0910e2 >2014-04-29 16:05:00,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] Failed in GetHardwareInfoVDS method >2014-04-29 16:05:00,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:05:00,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@253395a9 >2014-04-29 16:05:00,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] HostName = 10.34.27.209 >2014-04-29 16:05:00,884 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:05:00,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [1f57e74f] FINISH, GetHardwareInfoVDSCommand, log id: 5e0910e2 >2014-04-29 16:05:00,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [1f57e74f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:05:00,901 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [75b9c683] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:05:00,902 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [44666754] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:05:00,903 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [44666754] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 57b38b47 >2014-04-29 16:05:00,917 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [44666754] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:05:00,918 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [44666754] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:05:01,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6f65ee60 >2014-04-29 16:05:01,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] Failed in GetHardwareInfoVDS method >2014-04-29 16:05:01,974 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:05:01,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@d0519c3 >2014-04-29 16:05:01,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] HostName = 10.34.27.209 >2014-04-29 16:05:01,975 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:05:01,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] FINISH, GetHardwareInfoVDSCommand, log id: 6f65ee60 >2014-04-29 16:05:01,988 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:05:01,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [6f3dff0c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:05:02,221 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-90) [68247418] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:05:02,223 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-90) [70857f55] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:05:02,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-90) [70857f55] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6cc4ebc5 >2014-04-29 16:08:02,231 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-90) [70857f55] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:08:02,231 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-90) [70857f55] FINISH, ConnectStorageServerVDSCommand, log id: 6cc4ebc5 >2014-04-29 16:08:02,231 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-90) [70857f55] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:08:02,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 73f4f4e1 >2014-04-29 16:08:02,676 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 73f4f4e1 >2014-04-29 16:08:02,676 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:08:02,687 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-90) [53eebb47] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:08:02,688 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [53eebb47] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 40f2ad23 >2014-04-29 16:08:02,691 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [53eebb47] FINISH, SetVdsStatusVDSCommand, log id: 40f2ad23 >2014-04-29 16:08:02,697 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [53eebb47] Correlation ID: 53eebb47, Job ID: 4c21c668-3e35-400a-9b39-4c4443a1b477, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:08:02,703 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [53eebb47] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:08:02,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [53eebb47] Correlation ID: 68247418, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:08:02,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-90) [56712416] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:08:02,709 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-90) [23747c2a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:08:02,710 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) [23747c2a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:10:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [62a73d1c] Autorecovering 1 hosts >2014-04-29 16:10:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [62a73d1c] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:10:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:10:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:10:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:10:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3d4338c6 >2014-04-29 16:10:00,025 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] FINISH, SetVdsStatusVDSCommand, log id: 3d4338c6 >2014-04-29 16:10:00,026 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1c362a86 >2014-04-29 16:10:00,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 41be86c8 >2014-04-29 16:10:00,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] Failed in GetHardwareInfoVDS method >2014-04-29 16:10:00,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:10:00,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4382d630 >2014-04-29 16:10:00,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] HostName = 10.34.27.209 >2014-04-29 16:10:00,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:10:00,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [39cb1b49] FINISH, GetHardwareInfoVDSCommand, log id: 41be86c8 >2014-04-29 16:10:00,891 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [39cb1b49] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:10:00,900 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-7) [7f2ad3eb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:10:00,901 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [2c700f3a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:10:00,902 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [2c700f3a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1c362a86 >2014-04-29 16:10:00,920 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [2c700f3a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:10:00,921 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [2c700f3a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:10:03,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [4fd42b14] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 417d212d >2014-04-29 16:10:03,303 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [4fd42b14] Failed in GetHardwareInfoVDS method >2014-04-29 16:10:03,304 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [4fd42b14] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:10:03,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [4fd42b14] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7e0d32f1 >2014-04-29 16:10:03,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [4fd42b14] HostName = 10.34.27.209 >2014-04-29 16:10:03,304 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [4fd42b14] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:10:03,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [4fd42b14] FINISH, GetHardwareInfoVDSCommand, log id: 417d212d >2014-04-29 16:10:03,309 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [4fd42b14] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:10:03,319 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [4fd42b14] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:10:03,542 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-21) [2adf98a3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:10:03,543 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-21) [2868b44] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:10:03,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [2868b44] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 32914565 >2014-04-29 16:13:03,551 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [2868b44] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:13:03,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [2868b44] FINISH, ConnectStorageServerVDSCommand, log id: 32914565 >2014-04-29 16:13:03,551 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-21) [2868b44] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:13:03,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 434883e7 >2014-04-29 16:13:03,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 434883e7 >2014-04-29 16:13:03,999 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:13:04,010 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-21) [27437e9a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:13:04,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [27437e9a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 446b23a1 >2014-04-29 16:13:04,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [27437e9a] FINISH, SetVdsStatusVDSCommand, log id: 446b23a1 >2014-04-29 16:13:04,019 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [27437e9a] Correlation ID: 27437e9a, Job ID: 79149470-2772-45a9-b798-f9dd5a7c72df, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:13:04,027 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [27437e9a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:13:04,030 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [27437e9a] Correlation ID: 2adf98a3, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:13:04,034 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-21) [20b7830e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:13:04,053 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-21) [5c279649] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:13:04,054 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) [5c279649] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) [439a91a9] Autorecovering 1 hosts >2014-04-29 16:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-25) [439a91a9] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4d0e9634 >2014-04-29 16:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] FINISH, SetVdsStatusVDSCommand, log id: 4d0e9634 >2014-04-29 16:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 15124a90 >2014-04-29 16:15:00,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 113e63e9 >2014-04-29 16:15:00,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] Failed in GetHardwareInfoVDS method >2014-04-29 16:15:00,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:15:00,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@710f0a48 >2014-04-29 16:15:00,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] HostName = 10.34.27.209 >2014-04-29 16:15:00,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:15:00,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7fbd1496] FINISH, GetHardwareInfoVDSCommand, log id: 113e63e9 >2014-04-29 16:15:00,826 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [7fbd1496] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:15:00,835 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-25) [4adc056e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:15:00,836 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:15:00,836 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 15124a90 >2014-04-29 16:15:00,850 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:15:00,851 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:15:01,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 134c169b >2014-04-29 16:15:01,628 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] Failed in GetHardwareInfoVDS method >2014-04-29 16:15:01,628 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:15:01,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@10ac7459 >2014-04-29 16:15:01,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] HostName = 10.34.27.209 >2014-04-29 16:15:01,628 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:15:01,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [3b184600] FINISH, GetHardwareInfoVDSCommand, log id: 134c169b >2014-04-29 16:15:01,633 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [3b184600] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:15:01,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [3b184600] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:15:01,868 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-75) [5d8a34ce] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:15:01,870 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-75) [76535c12] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:15:01,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-75) [76535c12] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2ca417cb >2014-04-29 16:18:01,880 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-75) [76535c12] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:18:01,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-75) [76535c12] FINISH, ConnectStorageServerVDSCommand, log id: 2ca417cb >2014-04-29 16:18:01,880 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-75) [76535c12] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:18:01,885 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 71f9d0c2 >2014-04-29 16:18:02,351 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 71f9d0c2 >2014-04-29 16:18:02,352 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:18:02,362 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-75) [3ea8e3ea] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:18:02,363 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [3ea8e3ea] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 588c78d8 >2014-04-29 16:18:02,366 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [3ea8e3ea] FINISH, SetVdsStatusVDSCommand, log id: 588c78d8 >2014-04-29 16:18:02,372 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [3ea8e3ea] Correlation ID: 3ea8e3ea, Job ID: c15d09b9-3ee4-4586-8eaa-c88ea519e3e7, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:18:02,380 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [3ea8e3ea] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:18:02,382 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [3ea8e3ea] Correlation ID: 5d8a34ce, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:18:02,386 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-75) [3c1d3fe4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:18:02,387 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-75) [2b91736e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:18:02,390 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-75) [2b91736e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) [52e4e6d5] Autorecovering 1 hosts >2014-04-29 16:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) [52e4e6d5] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 59524987 >2014-04-29 16:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] FINISH, SetVdsStatusVDSCommand, log id: 59524987 >2014-04-29 16:20:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 55df2e65 >2014-04-29 16:20:00,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 72ac1c75 >2014-04-29 16:20:00,935 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] Failed in GetHardwareInfoVDS method >2014-04-29 16:20:00,936 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:20:00,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1808024 >2014-04-29 16:20:00,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] HostName = 10.34.27.209 >2014-04-29 16:20:00,936 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:20:00,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [7617a4e8] FINISH, GetHardwareInfoVDSCommand, log id: 72ac1c75 >2014-04-29 16:20:00,941 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [7617a4e8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:20:00,950 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [ddbcd8f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:20:00,951 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [5628a0bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:20:00,952 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [5628a0bc] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 55df2e65 >2014-04-29 16:20:00,967 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [5628a0bc] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:20:00,968 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [5628a0bc] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:20:02,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 44a6ffb6 >2014-04-29 16:20:03,009 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] Failed in GetHardwareInfoVDS method >2014-04-29 16:20:03,010 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:20:03,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2fde28f0 >2014-04-29 16:20:03,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] HostName = 10.34.27.209 >2014-04-29 16:20:03,010 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:20:03,010 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [358b4af5] FINISH, GetHardwareInfoVDSCommand, log id: 44a6ffb6 >2014-04-29 16:20:03,038 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [358b4af5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:20:03,067 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [358b4af5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:20:03,305 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-10) [262c8e54] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:20:03,307 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-10) [2dfa6de3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:20:03,314 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-10) [2dfa6de3] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 537055d8 >2014-04-29 16:23:03,316 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-10) [2dfa6de3] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:23:03,316 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-10) [2dfa6de3] FINISH, ConnectStorageServerVDSCommand, log id: 537055d8 >2014-04-29 16:23:03,316 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-10) [2dfa6de3] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:23:03,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 2566d012 >2014-04-29 16:23:03,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 2566d012 >2014-04-29 16:23:03,773 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:23:03,786 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-10) [3680782c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:23:03,788 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [3680782c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 477f3991 >2014-04-29 16:23:03,791 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [3680782c] FINISH, SetVdsStatusVDSCommand, log id: 477f3991 >2014-04-29 16:23:03,797 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [3680782c] Correlation ID: 3680782c, Job ID: 58a7857b-de4b-4b96-8579-fcacf3d7bf06, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:23:03,803 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [3680782c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:23:03,805 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [3680782c] Correlation ID: 262c8e54, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:23:03,811 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-10) [22aa797a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:23:03,813 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:23:03,814 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-55) [e3e34a3] Autorecovering 1 hosts >2014-04-29 16:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-55) [e3e34a3] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7246b337 >2014-04-29 16:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] FINISH, SetVdsStatusVDSCommand, log id: 7246b337 >2014-04-29 16:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 451cec95 >2014-04-29 16:25:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 29f43666 >2014-04-29 16:25:00,866 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] Failed in GetHardwareInfoVDS method >2014-04-29 16:25:00,866 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:25:00,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@601f780e >2014-04-29 16:25:00,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] HostName = 10.34.27.209 >2014-04-29 16:25:00,866 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:25:00,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [2ef76b32] FINISH, GetHardwareInfoVDSCommand, log id: 29f43666 >2014-04-29 16:25:00,871 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [2ef76b32] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:25:00,881 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [138ca230] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:25:00,882 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:25:00,883 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 451cec95 >2014-04-29 16:25:00,899 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:25:00,900 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:25:04,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: a955a6e >2014-04-29 16:25:04,392 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] Failed in GetHardwareInfoVDS method >2014-04-29 16:25:04,392 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:25:04,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5363f12f >2014-04-29 16:25:04,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] HostName = 10.34.27.209 >2014-04-29 16:25:04,392 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:25:04,392 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [c20abc] FINISH, GetHardwareInfoVDSCommand, log id: a955a6e >2014-04-29 16:25:04,400 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [c20abc] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:25:04,410 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [c20abc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:25:04,632 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-16) [28086b09] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:25:04,634 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-16) [41886f9d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:25:04,640 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-16) [41886f9d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 109cd700 >2014-04-29 16:28:04,641 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-16) [41886f9d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:28:04,641 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-16) [41886f9d] FINISH, ConnectStorageServerVDSCommand, log id: 109cd700 >2014-04-29 16:28:04,641 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-16) [41886f9d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:28:04,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5606622c >2014-04-29 16:28:05,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5606622c >2014-04-29 16:28:05,094 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:28:05,106 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-16) [7236bce7] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:28:05,107 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [7236bce7] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 50d057fa >2014-04-29 16:28:05,109 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [7236bce7] FINISH, SetVdsStatusVDSCommand, log id: 50d057fa >2014-04-29 16:28:05,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [7236bce7] Correlation ID: 7236bce7, Job ID: a184eef6-8f65-40a2-b752-b7305632ed3b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:28:05,121 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [7236bce7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:28:05,123 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [7236bce7] Correlation ID: 28086b09, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:28:05,127 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-16) [40b65a41] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:28:05,128 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-16) [51c90391] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:28:05,129 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) [51c90391] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-52) [35198fe3] Autorecovering 1 hosts >2014-04-29 16:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-52) [35198fe3] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4537d92c >2014-04-29 16:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] FINISH, SetVdsStatusVDSCommand, log id: 4537d92c >2014-04-29 16:30:00,018 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 613a005 >2014-04-29 16:30:00,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7a6c90d7 >2014-04-29 16:30:00,885 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] Failed in GetHardwareInfoVDS method >2014-04-29 16:30:00,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:30:00,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6d99520b >2014-04-29 16:30:00,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] HostName = 10.34.27.209 >2014-04-29 16:30:00,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:30:00,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [b44aba4] FINISH, GetHardwareInfoVDSCommand, log id: 7a6c90d7 >2014-04-29 16:30:00,921 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [b44aba4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:30:00,933 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-52) [178c2146] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:30:00,934 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:30:00,935 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 613a005 >2014-04-29 16:30:00,953 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:30:00,955 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:30:02,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 20659968 >2014-04-29 16:30:02,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) Failed in GetHardwareInfoVDS method >2014-04-29 16:30:02,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:30:02,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6e3a8b75 >2014-04-29 16:30:02,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) HostName = 10.34.27.209 >2014-04-29 16:30:02,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:30:02,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) FINISH, GetHardwareInfoVDSCommand, log id: 20659968 >2014-04-29 16:30:02,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:30:02,785 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:30:03,008 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-35) [18b42e76] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:30:03,009 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-35) [77442c58] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:30:03,016 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-35) [77442c58] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 300ceb0d >2014-04-29 16:33:03,020 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-35) [77442c58] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:33:03,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-35) [77442c58] FINISH, ConnectStorageServerVDSCommand, log id: 300ceb0d >2014-04-29 16:33:03,021 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-35) [77442c58] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:33:03,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5738a8f1 >2014-04-29 16:33:03,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5738a8f1 >2014-04-29 16:33:03,477 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:33:03,489 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-35) [2b9d0e80] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:33:03,490 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [2b9d0e80] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 733a9d1 >2014-04-29 16:33:03,493 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [2b9d0e80] FINISH, SetVdsStatusVDSCommand, log id: 733a9d1 >2014-04-29 16:33:03,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [2b9d0e80] Correlation ID: 2b9d0e80, Job ID: 48e5cfb6-d6f9-4055-923c-fece9717db29, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:33:03,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [2b9d0e80] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:33:03,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [2b9d0e80] Correlation ID: 18b42e76, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:33:03,534 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-35) [77fb9a35] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:33:03,535 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-35) [3b027fc1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:33:03,536 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [3b027fc1] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [2c700f3a] Autorecovering 1 hosts >2014-04-29 16:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [2c700f3a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 8d3d48a >2014-04-29 16:35:00,032 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] FINISH, SetVdsStatusVDSCommand, log id: 8d3d48a >2014-04-29 16:35:00,034 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4db154ce >2014-04-29 16:35:00,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 412443e0 >2014-04-29 16:35:00,901 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] Failed in GetHardwareInfoVDS method >2014-04-29 16:35:00,901 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:35:00,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@bf725d2 >2014-04-29 16:35:00,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] HostName = 10.34.27.209 >2014-04-29 16:35:00,901 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:35:00,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [6d807d6c] FINISH, GetHardwareInfoVDSCommand, log id: 412443e0 >2014-04-29 16:35:00,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [6d807d6c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:35:00,926 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-7) [222ed399] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:35:00,927 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [22747472] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:35:00,928 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [22747472] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4db154ce >2014-04-29 16:35:00,942 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [22747472] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:35:00,943 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [22747472] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:35:03,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [60fc8501] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 24dfe28e >2014-04-29 16:35:04,124 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [60fc8501] Failed in GetHardwareInfoVDS method >2014-04-29 16:35:04,124 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [60fc8501] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:35:04,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [60fc8501] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@553cde04 >2014-04-29 16:35:04,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [60fc8501] HostName = 10.34.27.209 >2014-04-29 16:35:04,124 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [60fc8501] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:35:04,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [60fc8501] FINISH, GetHardwareInfoVDSCommand, log id: 24dfe28e >2014-04-29 16:35:04,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [60fc8501] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:35:04,138 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [60fc8501] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:35:04,360 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-43) [4b6c72d5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:35:04,361 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [ae86582] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:35:04,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [ae86582] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 68f69898 >2014-04-29 16:36:04,995 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] Start refreshing all users data >2014-04-29 16:36:07,256 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update VMs/Templates Ovf. >2014-04-29 16:36:07,257 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update VM OVFs in Data Center NFS >2014-04-29 16:36:07,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully updated VM OVFs in Data Center NFS >2014-04-29 16:36:07,261 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to update template OVFs in Data Center NFS >2014-04-29 16:36:07,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully updated templates OVFs in Data Center NFS >2014-04-29 16:36:07,262 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 16:36:07,263 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 16:38:04,369 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [ae86582] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:38:04,369 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [ae86582] FINISH, ConnectStorageServerVDSCommand, log id: 68f69898 >2014-04-29 16:38:04,369 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [ae86582] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:38:04,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 60887232 >2014-04-29 16:38:04,826 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 60887232 >2014-04-29 16:38:04,826 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:38:04,838 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-43) [795d7210] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:38:04,839 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [795d7210] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 39e0fac >2014-04-29 16:38:04,841 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [795d7210] FINISH, SetVdsStatusVDSCommand, log id: 39e0fac >2014-04-29 16:38:04,848 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [795d7210] Correlation ID: 795d7210, Job ID: 5864f2c5-b96f-4f4d-8a0c-6e4e36294ba7, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:38:04,855 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [795d7210] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:38:04,857 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [795d7210] Correlation ID: 4b6c72d5, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:38:04,861 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [109e0071] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:38:04,862 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [137577be] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:38:04,863 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [137577be] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-44) [34a5a5ec] Autorecovering 1 hosts >2014-04-29 16:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-44) [34a5a5ec] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3e8947af >2014-04-29 16:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] FINISH, SetVdsStatusVDSCommand, log id: 3e8947af >2014-04-29 16:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 73f90e45 >2014-04-29 16:40:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 53fc8645 >2014-04-29 16:40:00,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] Failed in GetHardwareInfoVDS method >2014-04-29 16:40:00,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:40:00,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@67d34716 >2014-04-29 16:40:00,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] HostName = 10.34.27.209 >2014-04-29 16:40:00,878 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:40:00,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [52a6013f] FINISH, GetHardwareInfoVDSCommand, log id: 53fc8645 >2014-04-29 16:40:00,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [52a6013f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:40:00,890 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [363a9e6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:40:00,891 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:40:00,892 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 73f90e45 >2014-04-29 16:40:00,904 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:40:00,905 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:40:02,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 12f356ff >2014-04-29 16:40:02,454 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] Failed in GetHardwareInfoVDS method >2014-04-29 16:40:02,454 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:40:02,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7faa1907 >2014-04-29 16:40:02,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] HostName = 10.34.27.209 >2014-04-29 16:40:02,454 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:40:02,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] FINISH, GetHardwareInfoVDSCommand, log id: 12f356ff >2014-04-29 16:40:02,460 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:40:02,474 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [6f94e9f8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:40:02,698 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-29) [4c6270cd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:40:02,700 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [1339d158] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:40:02,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [1339d158] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4601bad1 >2014-04-29 16:43:02,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [1339d158] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:43:02,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [1339d158] FINISH, ConnectStorageServerVDSCommand, log id: 4601bad1 >2014-04-29 16:43:02,708 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [1339d158] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:43:02,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 439df327 >2014-04-29 16:43:03,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 439df327 >2014-04-29 16:43:03,155 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:43:03,166 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-29) [5ee5ee5a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:43:03,167 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [5ee5ee5a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6c5eb45a >2014-04-29 16:43:03,170 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [5ee5ee5a] FINISH, SetVdsStatusVDSCommand, log id: 6c5eb45a >2014-04-29 16:43:03,176 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [5ee5ee5a] Correlation ID: 5ee5ee5a, Job ID: 5d97c912-157d-4a69-84e2-079f29fe4ca0, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:43:03,184 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [5ee5ee5a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:43:03,186 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [5ee5ee5a] Correlation ID: 4c6270cd, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:43:03,189 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-29) [7c8feec6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:43:03,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-29) [15eddc0c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:43:03,191 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) [15eddc0c] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) [7a93afa1] Autorecovering 1 hosts >2014-04-29 16:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) [7a93afa1] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 41130c22 >2014-04-29 16:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] FINISH, SetVdsStatusVDSCommand, log id: 41130c22 >2014-04-29 16:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2f864619 >2014-04-29 16:45:00,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 52b84169 >2014-04-29 16:45:00,874 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] Failed in GetHardwareInfoVDS method >2014-04-29 16:45:00,874 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:45:00,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3ece7eec >2014-04-29 16:45:00,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] HostName = 10.34.27.209 >2014-04-29 16:45:00,874 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:45:00,874 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [6c62233] FINISH, GetHardwareInfoVDSCommand, log id: 52b84169 >2014-04-29 16:45:00,880 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [6c62233] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:45:00,888 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [2d82a38a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:45:00,889 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:45:00,890 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2f864619 >2014-04-29 16:45:00,903 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:45:00,904 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:45:03,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [82758c6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4e89508a >2014-04-29 16:45:03,798 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [82758c6] Failed in GetHardwareInfoVDS method >2014-04-29 16:45:03,799 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [82758c6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:45:03,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [82758c6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6a8a729d >2014-04-29 16:45:03,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [82758c6] HostName = 10.34.27.209 >2014-04-29 16:45:03,799 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [82758c6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:45:03,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [82758c6] FINISH, GetHardwareInfoVDSCommand, log id: 4e89508a >2014-04-29 16:45:03,803 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [82758c6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:45:03,812 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [82758c6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:45:04,040 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-82) [79d5b1d4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:45:04,041 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-82) [4bd0f185] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:45:04,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [4bd0f185] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 64b01fcc >2014-04-29 16:48:04,050 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [4bd0f185] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:48:04,050 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [4bd0f185] FINISH, ConnectStorageServerVDSCommand, log id: 64b01fcc >2014-04-29 16:48:04,050 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-82) [4bd0f185] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:48:04,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 223ceee7 >2014-04-29 16:48:04,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 223ceee7 >2014-04-29 16:48:04,502 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:48:04,515 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-82) [6ffd7a1c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:48:04,517 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [6ffd7a1c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5096c327 >2014-04-29 16:48:04,520 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [6ffd7a1c] FINISH, SetVdsStatusVDSCommand, log id: 5096c327 >2014-04-29 16:48:04,536 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [6ffd7a1c] Correlation ID: 6ffd7a1c, Job ID: 3723db00-408e-4a43-8ec2-769080e0ae16, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:48:04,550 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [6ffd7a1c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:48:04,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [6ffd7a1c] Correlation ID: 79d5b1d4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:48:04,576 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-82) [41a6eb4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:48:04,577 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-82) [108d9d32] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:48:04,578 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) [108d9d32] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:48:21,940 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [22747472] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space.storage domain has 3 GB of free space >2014-04-29 16:50:00,009 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) [67a1f47c] Autorecovering 1 hosts >2014-04-29 16:50:00,009 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) [67a1f47c] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:50:00,011 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:50:00,011 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:50:00,011 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:50:00,011 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:50:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 333ccf25 >2014-04-29 16:50:00,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] FINISH, SetVdsStatusVDSCommand, log id: 333ccf25 >2014-04-29 16:50:00,019 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 731f1257 >2014-04-29 16:50:00,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 33f6c342 >2014-04-29 16:50:00,899 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] Failed in GetHardwareInfoVDS method >2014-04-29 16:50:00,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:50:00,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@d8d3cb5 >2014-04-29 16:50:00,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] HostName = 10.34.27.209 >2014-04-29 16:50:00,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:50:00,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [62b10c31] FINISH, GetHardwareInfoVDSCommand, log id: 33f6c342 >2014-04-29 16:50:00,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [62b10c31] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:50:00,917 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [544c5e2c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:50:00,918 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:50:00,919 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 731f1257 >2014-04-29 16:50:00,934 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:50:00,935 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:50:01,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 44b7c976 >2014-04-29 16:50:02,162 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] Failed in GetHardwareInfoVDS method >2014-04-29 16:50:02,162 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:50:02,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7e0b8538 >2014-04-29 16:50:02,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] HostName = 10.34.27.209 >2014-04-29 16:50:02,164 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:50:02,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [79f533ba] FINISH, GetHardwareInfoVDSCommand, log id: 44b7c976 >2014-04-29 16:50:02,172 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [79f533ba] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:50:02,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [79f533ba] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:50:02,407 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [32e2f6e1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:50:02,409 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [6f3af511] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:50:02,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [6f3af511] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2e6b513a >2014-04-29 16:53:02,421 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [6f3af511] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:53:02,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [6f3af511] FINISH, ConnectStorageServerVDSCommand, log id: 2e6b513a >2014-04-29 16:53:02,422 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [6f3af511] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:53:02,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 52514b86 >2014-04-29 16:53:02,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 52514b86 >2014-04-29 16:53:02,875 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:53:02,888 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-81) [6ade72] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:53:02,889 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [6ade72] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 63bdbf80 >2014-04-29 16:53:02,893 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [6ade72] FINISH, SetVdsStatusVDSCommand, log id: 63bdbf80 >2014-04-29 16:53:02,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [6ade72] Correlation ID: 6ade72, Job ID: 20374b09-4456-400f-b69c-c8818295f453, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:53:02,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [6ade72] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:53:02,908 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [6ade72] Correlation ID: 32e2f6e1, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:53:02,911 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [20b7774d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:53:02,913 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [5223973b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:53:02,913 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [5223973b] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 16:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [43f50b33] Autorecovering 1 hosts >2014-04-29 16:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [43f50b33] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 16:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 16:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 16:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 43f9a07f >2014-04-29 16:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] FINISH, SetVdsStatusVDSCommand, log id: 43f9a07f >2014-04-29 16:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 727c6f87 >2014-04-29 16:55:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 749c38b8 >2014-04-29 16:55:00,903 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] Failed in GetHardwareInfoVDS method >2014-04-29 16:55:00,903 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:55:00,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3c4ff3d7 >2014-04-29 16:55:00,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] HostName = 10.34.27.209 >2014-04-29 16:55:00,903 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:55:00,903 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [4327a3fd] FINISH, GetHardwareInfoVDSCommand, log id: 749c38b8 >2014-04-29 16:55:00,908 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [4327a3fd] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:55:00,916 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [1c3b6564] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:55:00,917 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [4f262b05] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:55:00,918 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [4f262b05] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 727c6f87 >2014-04-29 16:55:00,931 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [4f262b05] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 16:55:00,932 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [4f262b05] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 16:55:03,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [790d05f6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 10864096 >2014-04-29 16:55:03,488 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [790d05f6] Failed in GetHardwareInfoVDS method >2014-04-29 16:55:03,489 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [790d05f6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:55:03,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [790d05f6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5dfa4053 >2014-04-29 16:55:03,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [790d05f6] HostName = 10.34.27.209 >2014-04-29 16:55:03,489 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [790d05f6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 16:55:03,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [790d05f6] FINISH, GetHardwareInfoVDSCommand, log id: 10864096 >2014-04-29 16:55:03,493 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [790d05f6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 16:55:03,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [790d05f6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 16:55:03,725 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-45) [15191d52] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:55:03,730 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [3b0c5116] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 16:55:03,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [3b0c5116] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5741d8c4 >2014-04-29 16:58:03,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [3b0c5116] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 16:58:03,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [3b0c5116] FINISH, ConnectStorageServerVDSCommand, log id: 5741d8c4 >2014-04-29 16:58:03,739 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [3b0c5116] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 16:58:03,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 2c3b107b >2014-04-29 16:58:04,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 2c3b107b >2014-04-29 16:58:04,226 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 16:58:04,237 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-45) [2dd7d65b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:58:04,238 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [2dd7d65b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 317e486e >2014-04-29 16:58:04,241 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [2dd7d65b] FINISH, SetVdsStatusVDSCommand, log id: 317e486e >2014-04-29 16:58:04,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [2dd7d65b] Correlation ID: 2dd7d65b, Job ID: c6a2795b-fe08-4595-bc75-9b9ee199fda7, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 16:58:04,253 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [2dd7d65b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 16:58:04,255 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [2dd7d65b] Correlation ID: 15191d52, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 16:58:04,260 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [53385300] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:58:04,261 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [204cd92a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 16:58:04,261 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) [204cd92a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) [1db304d2] Autorecovering 1 hosts >2014-04-29 17:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) [1db304d2] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1403581 >2014-04-29 17:00:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] FINISH, SetVdsStatusVDSCommand, log id: 1403581 >2014-04-29 17:00:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1ff567da >2014-04-29 17:00:00,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 625caf78 >2014-04-29 17:00:01,003 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] Failed in GetHardwareInfoVDS method >2014-04-29 17:00:01,003 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:00:01,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@633f9874 >2014-04-29 17:00:01,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] HostName = 10.34.27.209 >2014-04-29 17:00:01,003 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:00:01,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6d14c30b] FINISH, GetHardwareInfoVDSCommand, log id: 625caf78 >2014-04-29 17:00:01,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [6d14c30b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:00:01,018 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-24) [b3d1db8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:00:01,019 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:00:01,020 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1ff567da >2014-04-29 17:00:01,037 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:00:01,038 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:00:01,615 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [18da1d2c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7ad3eb0f >2014-04-29 17:00:01,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [18da1d2c] Failed in GetHardwareInfoVDS method >2014-04-29 17:00:01,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [18da1d2c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:00:01,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [18da1d2c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6eed64c8 >2014-04-29 17:00:01,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [18da1d2c] HostName = 10.34.27.209 >2014-04-29 17:00:01,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [18da1d2c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:00:01,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [18da1d2c] FINISH, GetHardwareInfoVDSCommand, log id: 7ad3eb0f >2014-04-29 17:00:01,850 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [18da1d2c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:00:01,859 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [18da1d2c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:00:02,083 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-22) [57890b61] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:00:02,085 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [62f4a16a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:00:02,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [62f4a16a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 20806fb5 >2014-04-29 17:03:02,095 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [62f4a16a] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:03:02,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [62f4a16a] FINISH, ConnectStorageServerVDSCommand, log id: 20806fb5 >2014-04-29 17:03:02,095 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [62f4a16a] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:03:02,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1fa3f574 >2014-04-29 17:03:02,552 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1fa3f574 >2014-04-29 17:03:02,552 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:03:02,563 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-22) [41eec11b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:03:02,564 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [41eec11b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7c0151f8 >2014-04-29 17:03:02,567 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [41eec11b] FINISH, SetVdsStatusVDSCommand, log id: 7c0151f8 >2014-04-29 17:03:02,574 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [41eec11b] Correlation ID: 41eec11b, Job ID: b329b131-190b-49ef-9d69-5c92a86384a0, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:03:02,583 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [41eec11b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:03:02,585 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [41eec11b] Correlation ID: 57890b61, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:03:02,592 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-22) [799c97c6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:03:02,593 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-22) [2d93263f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:03:02,594 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [2d93263f] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-57) [46196251] Autorecovering 1 hosts >2014-04-29 17:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-57) [46196251] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:05:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:05:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: fdcc381 >2014-04-29 17:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] FINISH, SetVdsStatusVDSCommand, log id: fdcc381 >2014-04-29 17:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7cec62f1 >2014-04-29 17:05:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 604d7bca >2014-04-29 17:05:00,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] Failed in GetHardwareInfoVDS method >2014-04-29 17:05:00,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:05:00,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5e842a86 >2014-04-29 17:05:00,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] HostName = 10.34.27.209 >2014-04-29 17:05:00,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:05:00,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [373953a1] FINISH, GetHardwareInfoVDSCommand, log id: 604d7bca >2014-04-29 17:05:00,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [373953a1] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:05:00,897 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [48f9dc7a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:05:00,898 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:05:00,899 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7cec62f1 >2014-04-29 17:05:00,913 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:05:00,913 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:05:02,950 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3de2be85 >2014-04-29 17:05:03,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] Failed in GetHardwareInfoVDS method >2014-04-29 17:05:03,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:05:03,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@24a7c975 >2014-04-29 17:05:03,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] HostName = 10.34.27.209 >2014-04-29 17:05:03,174 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:05:03,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [777c9bf8] FINISH, GetHardwareInfoVDSCommand, log id: 3de2be85 >2014-04-29 17:05:03,179 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [777c9bf8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:05:03,188 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [777c9bf8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:05:03,410 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-44) [39756ff4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:05:03,415 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [9a7cdd5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:05:03,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [9a7cdd5] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6f2a9dbf >2014-04-29 17:08:03,424 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [9a7cdd5] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:08:03,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [9a7cdd5] FINISH, ConnectStorageServerVDSCommand, log id: 6f2a9dbf >2014-04-29 17:08:03,424 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [9a7cdd5] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:08:03,430 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7a68c8ca >2014-04-29 17:08:03,884 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7a68c8ca >2014-04-29 17:08:03,884 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:08:03,895 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-44) [719107ed] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:08:03,896 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [719107ed] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: d13611d >2014-04-29 17:08:03,899 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [719107ed] FINISH, SetVdsStatusVDSCommand, log id: d13611d >2014-04-29 17:08:03,904 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [719107ed] Correlation ID: 719107ed, Job ID: c805f1ee-9723-4953-9e6e-8b3af8a24eed, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:08:03,914 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [719107ed] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:08:03,916 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [719107ed] Correlation ID: 39756ff4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:08:03,921 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [529d52be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:08:03,923 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [1fea9efb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:08:03,924 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) [1fea9efb] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-40) [10c242c0] Autorecovering 1 hosts >2014-04-29 17:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-40) [10c242c0] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6e56c96f >2014-04-29 17:10:00,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] FINISH, SetVdsStatusVDSCommand, log id: 6e56c96f >2014-04-29 17:10:00,022 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 167e6490 >2014-04-29 17:10:00,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 23c19089 >2014-04-29 17:10:00,890 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] Failed in GetHardwareInfoVDS method >2014-04-29 17:10:00,891 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:10:00,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@78f824b5 >2014-04-29 17:10:00,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] HostName = 10.34.27.209 >2014-04-29 17:10:00,891 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:10:00,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [443d17be] FINISH, GetHardwareInfoVDSCommand, log id: 23c19089 >2014-04-29 17:10:00,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [443d17be] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:10:00,906 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-40) [1d4ce293] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:10:00,907 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:10:00,908 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 167e6490 >2014-04-29 17:10:00,928 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:10:00,929 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:10:01,278 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [43910787] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 8750243 >2014-04-29 17:10:01,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [43910787] Failed in GetHardwareInfoVDS method >2014-04-29 17:10:01,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [43910787] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:10:01,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [43910787] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@10ca2979 >2014-04-29 17:10:01,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [43910787] HostName = 10.34.27.209 >2014-04-29 17:10:01,501 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [43910787] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:10:01,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [43910787] FINISH, GetHardwareInfoVDSCommand, log id: 8750243 >2014-04-29 17:10:01,505 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [43910787] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:10:01,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [43910787] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:10:01,739 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-83) [4e89d1cb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:10:01,740 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-83) [15507c25] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:10:01,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-83) [15507c25] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 79c60e61 >2014-04-29 17:13:01,751 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-83) [15507c25] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:13:01,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-83) [15507c25] FINISH, ConnectStorageServerVDSCommand, log id: 79c60e61 >2014-04-29 17:13:01,751 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-83) [15507c25] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:13:01,754 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5e5bc648 >2014-04-29 17:13:02,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5e5bc648 >2014-04-29 17:13:02,201 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:13:02,214 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-83) [12ff6170] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:13:02,216 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [12ff6170] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7df9655d >2014-04-29 17:13:02,220 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [12ff6170] FINISH, SetVdsStatusVDSCommand, log id: 7df9655d >2014-04-29 17:13:02,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [12ff6170] Correlation ID: 12ff6170, Job ID: af0a3309-3d5a-47da-97ad-c53d289647f2, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:13:02,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [12ff6170] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:13:02,235 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [12ff6170] Correlation ID: 4e89d1cb, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:13:02,241 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-83) [55116359] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:13:02,242 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-83) [5473ac4a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:13:02,243 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-83) [5473ac4a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-63) Autorecovering 1 hosts >2014-04-29 17:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-63) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 77de19cc >2014-04-29 17:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] FINISH, SetVdsStatusVDSCommand, log id: 77de19cc >2014-04-29 17:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 50ef06f0 >2014-04-29 17:15:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 21f05999 >2014-04-29 17:15:00,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] Failed in GetHardwareInfoVDS method >2014-04-29 17:15:00,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:15:00,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7bc88ab5 >2014-04-29 17:15:00,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] HostName = 10.34.27.209 >2014-04-29 17:15:00,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:15:00,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [63f5268f] FINISH, GetHardwareInfoVDSCommand, log id: 21f05999 >2014-04-29 17:15:00,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [63f5268f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:15:00,885 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-63) [655f2988] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:15:00,886 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-63) [7335b154] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:15:00,886 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) [7335b154] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 50ef06f0 >2014-04-29 17:15:00,910 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [7335b154] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:15:00,911 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [7335b154] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:15:02,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1d5f961 >2014-04-29 17:15:02,829 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) Failed in GetHardwareInfoVDS method >2014-04-29 17:15:02,829 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:15:02,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6e75038 >2014-04-29 17:15:02,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) HostName = 10.34.27.209 >2014-04-29 17:15:02,829 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:15:02,829 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) FINISH, GetHardwareInfoVDSCommand, log id: 1d5f961 >2014-04-29 17:15:02,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:15:02,851 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:15:03,080 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-62) [2138f0e6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:15:03,082 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-62) [37b8f2a7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:15:03,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [37b8f2a7] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6e7cf0a8 >2014-04-29 17:18:03,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [37b8f2a7] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:18:03,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [37b8f2a7] FINISH, ConnectStorageServerVDSCommand, log id: 6e7cf0a8 >2014-04-29 17:18:03,090 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-62) [37b8f2a7] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:18:03,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 42d86850 >2014-04-29 17:18:03,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 42d86850 >2014-04-29 17:18:03,537 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:18:03,548 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-62) [3e73788f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:18:03,549 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [3e73788f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5749fd33 >2014-04-29 17:18:03,552 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [3e73788f] FINISH, SetVdsStatusVDSCommand, log id: 5749fd33 >2014-04-29 17:18:03,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [3e73788f] Correlation ID: 3e73788f, Job ID: 9e7307dc-d1d0-467c-a4cc-badc9d234afd, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:18:03,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [3e73788f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:18:03,566 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [3e73788f] Correlation ID: 2138f0e6, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:18:03,570 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-62) [2c687cd7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:18:03,571 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:18:03,572 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [4f262b05] Autorecovering 1 hosts >2014-04-29 17:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [4f262b05] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: cc90cc >2014-04-29 17:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] FINISH, SetVdsStatusVDSCommand, log id: cc90cc >2014-04-29 17:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 739f21a >2014-04-29 17:20:01,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2f7eb7b >2014-04-29 17:20:01,756 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] Failed in GetHardwareInfoVDS method >2014-04-29 17:20:01,756 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:20:01,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@794e6139 >2014-04-29 17:20:01,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] HostName = 10.34.27.209 >2014-04-29 17:20:01,756 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:20:01,756 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [28b0c258] FINISH, GetHardwareInfoVDSCommand, log id: 2f7eb7b >2014-04-29 17:20:01,760 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [28b0c258] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:20:01,769 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [7a87f9f2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:20:01,770 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [3e5346e2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:20:01,770 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [3e5346e2] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 739f21a >2014-04-29 17:20:01,793 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [3e5346e2] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:20:01,794 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [3e5346e2] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:20:03,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4dfcbcb9 >2014-04-29 17:20:04,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] Failed in GetHardwareInfoVDS method >2014-04-29 17:20:04,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:20:04,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@bca3daf >2014-04-29 17:20:04,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] HostName = 10.34.27.209 >2014-04-29 17:20:04,172 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:20:04,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [4a56abc9] FINISH, GetHardwareInfoVDSCommand, log id: 4dfcbcb9 >2014-04-29 17:20:04,177 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [4a56abc9] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:20:04,188 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [4a56abc9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:20:04,411 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [55aeb41] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:20:04,413 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [11ba0b66] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:20:04,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [11ba0b66] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50cb33e >2014-04-29 17:23:04,420 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [11ba0b66] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:23:04,421 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [11ba0b66] FINISH, ConnectStorageServerVDSCommand, log id: 50cb33e >2014-04-29 17:23:04,421 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [11ba0b66] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:23:04,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7a9434d6 >2014-04-29 17:23:04,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 7a9434d6 >2014-04-29 17:23:04,869 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:23:04,882 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-86) [362c8190] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:23:04,883 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [362c8190] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7fc67f6d >2014-04-29 17:23:04,886 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [362c8190] FINISH, SetVdsStatusVDSCommand, log id: 7fc67f6d >2014-04-29 17:23:04,893 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [362c8190] Correlation ID: 362c8190, Job ID: c48588f8-7066-46a8-bd1b-445ac8fdeb12, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:23:04,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [362c8190] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:23:04,902 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [362c8190] Correlation ID: 55aeb41, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:23:04,906 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [5f939d2e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:23:04,907 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [52f20496] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:23:04,908 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [52f20496] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-4) Autorecovering 1 hosts >2014-04-29 17:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-4) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 37786b9e >2014-04-29 17:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] FINISH, SetVdsStatusVDSCommand, log id: 37786b9e >2014-04-29 17:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 27f7deba >2014-04-29 17:25:00,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1cfafa7e >2014-04-29 17:25:00,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] Failed in GetHardwareInfoVDS method >2014-04-29 17:25:00,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:25:00,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2153750f >2014-04-29 17:25:00,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] HostName = 10.34.27.209 >2014-04-29 17:25:00,877 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:25:00,877 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [2bba4411] FINISH, GetHardwareInfoVDSCommand, log id: 1cfafa7e >2014-04-29 17:25:00,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [2bba4411] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:25:00,890 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-4) [89a68f7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:25:00,891 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-4) [56d5ef3f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:25:00,892 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [56d5ef3f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 27f7deba >2014-04-29 17:25:00,905 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [56d5ef3f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:25:00,906 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [56d5ef3f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:25:02,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 58b4e5fc >2014-04-29 17:25:02,489 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] Failed in GetHardwareInfoVDS method >2014-04-29 17:25:02,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:25:02,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4c6730e3 >2014-04-29 17:25:02,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] HostName = 10.34.27.209 >2014-04-29 17:25:02,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:25:02,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] FINISH, GetHardwareInfoVDSCommand, log id: 58b4e5fc >2014-04-29 17:25:02,495 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:25:02,503 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [3d8ff56f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:25:02,725 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-8) [66b86b1b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:25:02,727 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [355f73a3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:25:02,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [355f73a3] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 16d375a0 >2014-04-29 17:28:02,734 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [355f73a3] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:28:02,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [355f73a3] FINISH, ConnectStorageServerVDSCommand, log id: 16d375a0 >2014-04-29 17:28:02,734 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [355f73a3] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:28:02,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: adb1293 >2014-04-29 17:28:03,194 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: adb1293 >2014-04-29 17:28:03,194 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:28:03,204 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-8) [7fe1361c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:28:03,205 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [7fe1361c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 41a5e320 >2014-04-29 17:28:03,208 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [7fe1361c] FINISH, SetVdsStatusVDSCommand, log id: 41a5e320 >2014-04-29 17:28:03,214 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [7fe1361c] Correlation ID: 7fe1361c, Job ID: cd218694-481c-41bd-9a0b-ed8c4ba28a85, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:28:03,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [7fe1361c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:28:03,222 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [7fe1361c] Correlation ID: 66b86b1b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:28:03,226 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [35c53d83] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:28:03,227 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [5586a962] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:28:03,227 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [5586a962] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:30:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-75) [2b91736e] Autorecovering 1 hosts >2014-04-29 17:30:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-75) [2b91736e] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:30:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:30:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:30:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:30:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 8593621 >2014-04-29 17:30:00,020 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] FINISH, SetVdsStatusVDSCommand, log id: 8593621 >2014-04-29 17:30:00,026 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4b0a1c15 >2014-04-29 17:30:00,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 71cf12d5 >2014-04-29 17:30:00,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] Failed in GetHardwareInfoVDS method >2014-04-29 17:30:00,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:30:00,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@51dd9852 >2014-04-29 17:30:00,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] HostName = 10.34.27.209 >2014-04-29 17:30:00,886 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:30:00,886 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [70b3941e] FINISH, GetHardwareInfoVDSCommand, log id: 71cf12d5 >2014-04-29 17:30:00,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [70b3941e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:30:00,912 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-75) [1ce435ad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:30:00,913 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-75) [22863027] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:30:00,914 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) [22863027] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4b0a1c15 >2014-04-29 17:30:00,927 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [22863027] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:30:00,928 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [22863027] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:30:03,585 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 54a4156e >2014-04-29 17:30:03,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] Failed in GetHardwareInfoVDS method >2014-04-29 17:30:03,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:30:03,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@607b142b >2014-04-29 17:30:03,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] HostName = 10.34.27.209 >2014-04-29 17:30:03,808 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:30:03,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [570e542c] FINISH, GetHardwareInfoVDSCommand, log id: 54a4156e >2014-04-29 17:30:03,813 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [570e542c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:30:03,823 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [570e542c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:30:04,045 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-57) [76a689f7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:30:04,046 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [7e735fc9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:30:04,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [7e735fc9] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 18283e95 >2014-04-29 17:33:04,054 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [7e735fc9] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:33:04,054 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [7e735fc9] FINISH, ConnectStorageServerVDSCommand, log id: 18283e95 >2014-04-29 17:33:04,054 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [7e735fc9] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:33:04,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 39c64374 >2014-04-29 17:33:04,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 39c64374 >2014-04-29 17:33:04,511 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:33:04,523 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-57) [1be505a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:33:04,524 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [1be505a6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 54ce04cb >2014-04-29 17:33:04,527 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [1be505a6] FINISH, SetVdsStatusVDSCommand, log id: 54ce04cb >2014-04-29 17:33:04,532 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [1be505a6] Correlation ID: 1be505a6, Job ID: 203dfae7-2308-4a73-ad8f-f670bcf93f26, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:33:04,539 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [1be505a6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:33:04,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [1be505a6] Correlation ID: 76a689f7, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:33:04,544 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [f3a448e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:33:04,545 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [6627d833] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:33:04,546 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [6627d833] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-94) Autorecovering 1 hosts >2014-04-29 17:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-94) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 18f767e1 >2014-04-29 17:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] FINISH, SetVdsStatusVDSCommand, log id: 18f767e1 >2014-04-29 17:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1a83c314 >2014-04-29 17:35:01,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3e0a4041 >2014-04-29 17:35:01,901 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] Failed in GetHardwareInfoVDS method >2014-04-29 17:35:01,901 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:35:01,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@13008b39 >2014-04-29 17:35:01,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] HostName = 10.34.27.209 >2014-04-29 17:35:01,901 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:35:01,901 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4be83955] FINISH, GetHardwareInfoVDSCommand, log id: 3e0a4041 >2014-04-29 17:35:01,910 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [4be83955] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:35:01,919 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-94) [48a0d149] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:35:01,920 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-94) [181e5b1e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:35:01,921 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) [181e5b1e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1a83c314 >2014-04-29 17:35:01,935 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [181e5b1e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:35:01,936 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [181e5b1e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:35:05,191 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [213931a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4c0e7f09 >2014-04-29 17:35:05,521 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [213931a] Failed in GetHardwareInfoVDS method >2014-04-29 17:35:05,521 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [213931a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:35:05,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [213931a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4bb64ee1 >2014-04-29 17:35:05,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [213931a] HostName = 10.34.27.209 >2014-04-29 17:35:05,521 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [213931a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:35:05,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [213931a] FINISH, GetHardwareInfoVDSCommand, log id: 4c0e7f09 >2014-04-29 17:35:05,526 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [213931a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:35:05,536 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [213931a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:35:05,800 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-18) [55c9d494] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:35:05,802 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-18) [7922a475] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:35:05,808 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [7922a475] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 292ab88b >2014-04-29 17:36:04,997 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-5) [21573f3a] Start refreshing all users data >2014-04-29 17:36:07,263 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) [196fb384] Attempting to update VMs/Templates Ovf. >2014-04-29 17:36:07,264 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) [196fb384] Attempting to update VM OVFs in Data Center NFS >2014-04-29 17:36:07,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) [196fb384] Successfully updated VM OVFs in Data Center NFS >2014-04-29 17:36:07,265 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) [196fb384] Attempting to update template OVFs in Data Center NFS >2014-04-29 17:36:07,266 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) [196fb384] Successfully updated templates OVFs in Data Center NFS >2014-04-29 17:36:07,266 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) [196fb384] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 17:36:07,267 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-91) [196fb384] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 17:38:05,809 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [7922a475] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:38:05,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [7922a475] FINISH, ConnectStorageServerVDSCommand, log id: 292ab88b >2014-04-29 17:38:05,809 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-18) [7922a475] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:38:05,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3890f7de >2014-04-29 17:38:06,265 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 3890f7de >2014-04-29 17:38:06,265 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:38:06,284 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-18) [626e8d01] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:38:06,285 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [626e8d01] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5647bab2 >2014-04-29 17:38:06,288 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [626e8d01] FINISH, SetVdsStatusVDSCommand, log id: 5647bab2 >2014-04-29 17:38:06,294 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [626e8d01] Correlation ID: 626e8d01, Job ID: 0e44a421-0023-4488-9bc4-f4b953163cf2, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:38:06,301 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [626e8d01] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:38:06,303 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [626e8d01] Correlation ID: 55c9d494, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:38:06,309 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-18) [7ed4bdbc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:38:06,310 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-18) [3fe3bbb3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:38:06,310 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) [3fe3bbb3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-30) [1b655e9d] Autorecovering 1 hosts >2014-04-29 17:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-30) [1b655e9d] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [944d320] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [944d320] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [944d320] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [944d320] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [944d320] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 50fb509d >2014-04-29 17:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [944d320] FINISH, SetVdsStatusVDSCommand, log id: 50fb509d >2014-04-29 17:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) [944d320] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 341f1ce3 >2014-04-29 17:40:00,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [944d320] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 31836c45 >2014-04-29 17:40:01,049 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [944d320] Failed in GetHardwareInfoVDS method >2014-04-29 17:40:01,049 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [944d320] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:40:01,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [944d320] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2b393f68 >2014-04-29 17:40:01,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [944d320] HostName = 10.34.27.209 >2014-04-29 17:40:01,049 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [944d320] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:40:01,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [944d320] FINISH, GetHardwareInfoVDSCommand, log id: 31836c45 >2014-04-29 17:40:01,054 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [944d320] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:40:01,064 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-30) [5f8337a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:40:01,065 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-30) [67b98f68] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:40:01,066 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) [67b98f68] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 341f1ce3 >2014-04-29 17:40:01,079 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [67b98f68] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:40:01,080 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [67b98f68] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:40:03,788 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7a4d1b1c >2014-04-29 17:40:04,134 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] Failed in GetHardwareInfoVDS method >2014-04-29 17:40:04,134 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:40:04,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5c2f09fe >2014-04-29 17:40:04,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] HostName = 10.34.27.209 >2014-04-29 17:40:04,134 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:40:04,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [61943c16] FINISH, GetHardwareInfoVDSCommand, log id: 7a4d1b1c >2014-04-29 17:40:04,140 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [61943c16] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:40:04,149 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [61943c16] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:40:04,506 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-55) [38ac005b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:40:04,508 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [5fc44cc2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:40:04,515 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [5fc44cc2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 13095471 >2014-04-29 17:43:04,517 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [5fc44cc2] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:43:04,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [5fc44cc2] FINISH, ConnectStorageServerVDSCommand, log id: 13095471 >2014-04-29 17:43:04,517 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [5fc44cc2] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:43:04,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 482bbbb >2014-04-29 17:43:04,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 482bbbb >2014-04-29 17:43:04,964 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:43:04,982 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-55) [79882f09] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:43:04,983 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [79882f09] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3a73af54 >2014-04-29 17:43:04,986 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [79882f09] FINISH, SetVdsStatusVDSCommand, log id: 3a73af54 >2014-04-29 17:43:04,992 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [79882f09] Correlation ID: 79882f09, Job ID: 0f3f481e-bcb9-4dca-8571-572347dbe552, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:43:04,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [79882f09] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:43:05,000 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [79882f09] Correlation ID: 38ac005b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:43:05,004 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [703d54ac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:43:05,005 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [531e7ffa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:43:05,005 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [531e7ffa] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:45:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-76) [1238a5e6] Autorecovering 1 hosts >2014-04-29 17:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-76) [1238a5e6] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 311dd30a >2014-04-29 17:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] FINISH, SetVdsStatusVDSCommand, log id: 311dd30a >2014-04-29 17:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 43954c12 >2014-04-29 17:45:00,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1132bf05 >2014-04-29 17:45:00,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] Failed in GetHardwareInfoVDS method >2014-04-29 17:45:00,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:45:00,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1a9aea15 >2014-04-29 17:45:00,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] HostName = 10.34.27.209 >2014-04-29 17:45:00,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:45:00,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] FINISH, GetHardwareInfoVDSCommand, log id: 1132bf05 >2014-04-29 17:45:00,903 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [5d80f0b4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:45:00,913 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-76) [6fd918f9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:45:00,914 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-76) [4c2357c9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:45:00,915 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [4c2357c9] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 43954c12 >2014-04-29 17:45:00,929 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [4c2357c9] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:45:00,930 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [4c2357c9] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:45:02,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4a72873 >2014-04-29 17:45:02,592 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] Failed in GetHardwareInfoVDS method >2014-04-29 17:45:02,593 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:45:02,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7a509867 >2014-04-29 17:45:02,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] HostName = 10.34.27.209 >2014-04-29 17:45:02,593 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:45:02,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [5bd78af3] FINISH, GetHardwareInfoVDSCommand, log id: 4a72873 >2014-04-29 17:45:02,598 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [5bd78af3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:45:02,610 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [5bd78af3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:45:02,833 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-52) [50d526c5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:45:02,834 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-52) [53007c91] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:45:02,840 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [53007c91] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7a908ad5 >2014-04-29 17:48:02,842 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [53007c91] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:48:02,842 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [53007c91] FINISH, ConnectStorageServerVDSCommand, log id: 7a908ad5 >2014-04-29 17:48:02,842 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-52) [53007c91] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:48:02,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7a527223 >2014-04-29 17:48:03,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7a527223 >2014-04-29 17:48:03,287 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:48:03,297 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-52) [47b9bdab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:48:03,298 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [47b9bdab] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1d0e8320 >2014-04-29 17:48:03,301 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [47b9bdab] FINISH, SetVdsStatusVDSCommand, log id: 1d0e8320 >2014-04-29 17:48:03,306 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [47b9bdab] Correlation ID: 47b9bdab, Job ID: f54689a1-f676-4df5-9efa-5064bd1e4077, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:48:03,312 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [47b9bdab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:48:03,313 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [47b9bdab] Correlation ID: 50d526c5, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:48:03,317 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-52) [59849e5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:48:03,318 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-52) [54b31938] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:48:03,319 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [54b31938] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [22747472] Autorecovering 1 hosts >2014-04-29 17:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [22747472] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:50:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:50:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:50:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:50:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5334f2f7 >2014-04-29 17:50:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] FINISH, SetVdsStatusVDSCommand, log id: 5334f2f7 >2014-04-29 17:50:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1fbac8d0 >2014-04-29 17:50:00,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 201985e6 >2014-04-29 17:50:00,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] Failed in GetHardwareInfoVDS method >2014-04-29 17:50:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:50:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6ba5cf80 >2014-04-29 17:50:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] HostName = 10.34.27.209 >2014-04-29 17:50:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:50:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1e8597d6] FINISH, GetHardwareInfoVDSCommand, log id: 201985e6 >2014-04-29 17:50:00,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [1e8597d6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:50:00,750 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-7) [7a10586a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:50:00,751 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:50:00,752 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1fbac8d0 >2014-04-29 17:50:00,766 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:50:00,766 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:50:03,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [3a91960d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 44f2285b >2014-04-29 17:50:03,911 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [3a91960d] Failed in GetHardwareInfoVDS method >2014-04-29 17:50:03,911 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [3a91960d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:50:03,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [3a91960d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@763791b9 >2014-04-29 17:50:03,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [3a91960d] HostName = 10.34.27.209 >2014-04-29 17:50:03,911 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [3a91960d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:50:03,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [3a91960d] FINISH, GetHardwareInfoVDSCommand, log id: 44f2285b >2014-04-29 17:50:03,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [3a91960d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:50:03,926 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [3a91960d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:50:04,152 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-98) [21d51065] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:50:04,153 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [6c6e5010] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:50:04,160 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [6c6e5010] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7607d43a >2014-04-29 17:53:04,161 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [6c6e5010] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:53:04,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [6c6e5010] FINISH, ConnectStorageServerVDSCommand, log id: 7607d43a >2014-04-29 17:53:04,162 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [6c6e5010] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:53:04,166 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 13b7254c >2014-04-29 17:53:04,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 13b7254c >2014-04-29 17:53:04,614 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:53:04,629 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-98) [71928e38] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:53:04,630 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [71928e38] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3527a207 >2014-04-29 17:53:04,633 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [71928e38] FINISH, SetVdsStatusVDSCommand, log id: 3527a207 >2014-04-29 17:53:04,640 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [71928e38] Correlation ID: 71928e38, Job ID: bef40f3d-0490-44c4-9da5-a06e43e8de94, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:53:04,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [71928e38] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:53:04,651 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [71928e38] Correlation ID: 21d51065, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:53:04,655 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-98) [2abe5479] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:53:04,656 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-98) [26573b54] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:53:04,657 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [26573b54] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 17:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-18) [3fe3bbb3] Autorecovering 1 hosts >2014-04-29 17:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-18) [3fe3bbb3] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 17:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 17:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 17:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 932ca95 >2014-04-29 17:55:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] FINISH, SetVdsStatusVDSCommand, log id: 932ca95 >2014-04-29 17:55:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 227e448a >2014-04-29 17:55:00,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 54bd1e21 >2014-04-29 17:55:00,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] Failed in GetHardwareInfoVDS method >2014-04-29 17:55:00,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:55:00,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4c0533fd >2014-04-29 17:55:00,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] HostName = 10.34.27.209 >2014-04-29 17:55:00,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:55:00,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4dfcd856] FINISH, GetHardwareInfoVDSCommand, log id: 54bd1e21 >2014-04-29 17:55:00,719 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [4dfcd856] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:55:00,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-18) [10c7ccd1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:55:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-18) [4fd8ab38] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:55:00,731 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) [4fd8ab38] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 227e448a >2014-04-29 17:55:00,747 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4fd8ab38] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 17:55:00,748 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4fd8ab38] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 17:55:02,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [353e2b4d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 48d6543 >2014-04-29 17:55:02,403 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [353e2b4d] Failed in GetHardwareInfoVDS method >2014-04-29 17:55:02,403 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [353e2b4d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:55:02,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [353e2b4d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3ee0006f >2014-04-29 17:55:02,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [353e2b4d] HostName = 10.34.27.209 >2014-04-29 17:55:02,404 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [353e2b4d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 17:55:02,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [353e2b4d] FINISH, GetHardwareInfoVDSCommand, log id: 48d6543 >2014-04-29 17:55:02,433 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [353e2b4d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 17:55:02,484 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [353e2b4d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 17:55:02,706 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-53) [706f11d6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:55:02,708 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-53) [5c13d735] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 17:55:02,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [5c13d735] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5858fd88 >2014-04-29 17:58:02,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [5c13d735] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 17:58:02,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [5c13d735] FINISH, ConnectStorageServerVDSCommand, log id: 5858fd88 >2014-04-29 17:58:02,717 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-53) [5c13d735] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 17:58:02,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 784a2756 >2014-04-29 17:58:03,177 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 784a2756 >2014-04-29 17:58:03,177 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 17:58:03,208 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-53) [36ca77cd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:58:03,209 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [36ca77cd] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 57e40ac0 >2014-04-29 17:58:03,212 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [36ca77cd] FINISH, SetVdsStatusVDSCommand, log id: 57e40ac0 >2014-04-29 17:58:03,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [36ca77cd] Correlation ID: 36ca77cd, Job ID: e73e8a5b-6307-41b3-a04b-ccac919c68b3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 17:58:03,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [36ca77cd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 17:58:03,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [36ca77cd] Correlation ID: 706f11d6, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 17:58:03,232 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-53) [77d605eb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:58:03,233 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-53) [55817744] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 17:58:03,233 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) [55817744] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) [10a7c8e8] Autorecovering 1 hosts >2014-04-29 18:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) [10a7c8e8] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:00:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:00:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3832b9a2 >2014-04-29 18:00:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] FINISH, SetVdsStatusVDSCommand, log id: 3832b9a2 >2014-04-29 18:00:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 20778930 >2014-04-29 18:00:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 47017244 >2014-04-29 18:00:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] Failed in GetHardwareInfoVDS method >2014-04-29 18:00:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:00:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@57f97477 >2014-04-29 18:00:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] HostName = 10.34.27.209 >2014-04-29 18:00:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:00:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [66f74d4e] FINISH, GetHardwareInfoVDSCommand, log id: 47017244 >2014-04-29 18:00:00,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [66f74d4e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:00:00,725 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [381eabec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:00:00,726 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:00:00,727 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 20778930 >2014-04-29 18:00:00,754 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:00:00,755 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:00:03,590 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2c36f88a >2014-04-29 18:00:03,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] Failed in GetHardwareInfoVDS method >2014-04-29 18:00:03,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:00:03,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4cf742b3 >2014-04-29 18:00:03,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] HostName = 10.34.27.209 >2014-04-29 18:00:03,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:00:03,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [108addcd] FINISH, GetHardwareInfoVDSCommand, log id: 2c36f88a >2014-04-29 18:00:03,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [108addcd] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:00:03,829 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [108addcd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:00:04,059 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-50) [63ee6d59] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:00:04,061 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [4f44250d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:00:04,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [4f44250d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 797e4554 >2014-04-29 18:03:04,069 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [4f44250d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:03:04,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [4f44250d] FINISH, ConnectStorageServerVDSCommand, log id: 797e4554 >2014-04-29 18:03:04,070 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [4f44250d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:03:04,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 303efb96 >2014-04-29 18:03:04,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 303efb96 >2014-04-29 18:03:04,531 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:03:04,543 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-50) [12106455] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:03:04,544 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [12106455] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 697afa4f >2014-04-29 18:03:04,546 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [12106455] FINISH, SetVdsStatusVDSCommand, log id: 697afa4f >2014-04-29 18:03:04,552 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [12106455] Correlation ID: 12106455, Job ID: 8eaf63a7-7251-43c4-9d9f-449eff8900a2, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:03:04,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [12106455] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:03:04,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [12106455] Correlation ID: 63ee6d59, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:03:04,565 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [37369f4e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:03:04,566 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [5e6a378] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:03:04,567 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [5e6a378] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-63) [7335b154] Autorecovering 1 hosts >2014-04-29 18:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-63) [7335b154] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4aaa92c2 >2014-04-29 18:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] FINISH, SetVdsStatusVDSCommand, log id: 4aaa92c2 >2014-04-29 18:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 17ab6dd9 >2014-04-29 18:05:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 59bbde93 >2014-04-29 18:05:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] Failed in GetHardwareInfoVDS method >2014-04-29 18:05:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:05:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@76c9c100 >2014-04-29 18:05:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] HostName = 10.34.27.209 >2014-04-29 18:05:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:05:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [39fd7393] FINISH, GetHardwareInfoVDSCommand, log id: 59bbde93 >2014-04-29 18:05:00,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [39fd7393] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:05:00,721 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-63) [64421d46] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:05:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-63) [48b1d9c4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:05:00,723 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) [48b1d9c4] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 17ab6dd9 >2014-04-29 18:05:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [48b1d9c4] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:05:00,738 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [48b1d9c4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:05:01,925 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [23747c2a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 345e2ee7 >2014-04-29 18:05:02,147 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [23747c2a] Failed in GetHardwareInfoVDS method >2014-04-29 18:05:02,148 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [23747c2a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:05:02,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [23747c2a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5d78525a >2014-04-29 18:05:02,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [23747c2a] HostName = 10.34.27.209 >2014-04-29 18:05:02,148 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [23747c2a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:05:02,148 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [23747c2a] FINISH, GetHardwareInfoVDSCommand, log id: 345e2ee7 >2014-04-29 18:05:02,153 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [23747c2a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:05:02,163 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [23747c2a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:05:02,391 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-90) [6ea27e95] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:05:02,393 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-90) [776edf76] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:05:02,399 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-90) [776edf76] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 19112c50 >2014-04-29 18:08:02,401 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-90) [776edf76] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:08:02,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-90) [776edf76] FINISH, ConnectStorageServerVDSCommand, log id: 19112c50 >2014-04-29 18:08:02,401 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-90) [776edf76] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:08:02,404 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 22630d33 >2014-04-29 18:08:02,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 22630d33 >2014-04-29 18:08:02,854 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:08:02,864 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-90) [3bb313ae] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:08:02,865 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [3bb313ae] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: e780e42 >2014-04-29 18:08:02,868 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [3bb313ae] FINISH, SetVdsStatusVDSCommand, log id: e780e42 >2014-04-29 18:08:02,873 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [3bb313ae] Correlation ID: 3bb313ae, Job ID: 84e9c176-f350-49d7-aa38-a713148ea72a, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:08:02,880 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [3bb313ae] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:08:02,883 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [3bb313ae] Correlation ID: 6ea27e95, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:08:02,886 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-90) [10b05fde] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:08:02,888 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-90) [1d0ba8da] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:08:02,888 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-90) [1d0ba8da] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-93) Autorecovering 1 hosts >2014-04-29 18:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-93) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 365a7212 >2014-04-29 18:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] FINISH, SetVdsStatusVDSCommand, log id: 365a7212 >2014-04-29 18:10:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6752532c >2014-04-29 18:10:00,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 73ab4349 >2014-04-29 18:10:00,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] Failed in GetHardwareInfoVDS method >2014-04-29 18:10:00,772 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:10:00,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@70cea8f2 >2014-04-29 18:10:00,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] HostName = 10.34.27.209 >2014-04-29 18:10:00,772 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:10:00,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6632dff6] FINISH, GetHardwareInfoVDSCommand, log id: 73ab4349 >2014-04-29 18:10:00,778 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [6632dff6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:10:00,792 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [276a7510] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:10:00,793 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:10:00,794 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6752532c >2014-04-29 18:10:00,813 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:10:00,814 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:10:03,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 45f2f1ac >2014-04-29 18:10:03,468 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) Failed in GetHardwareInfoVDS method >2014-04-29 18:10:03,469 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:10:03,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@50d87714 >2014-04-29 18:10:03,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) HostName = 10.34.27.209 >2014-04-29 18:10:03,469 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:10:03,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) FINISH, GetHardwareInfoVDSCommand, log id: 45f2f1ac >2014-04-29 18:10:03,474 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:10:03,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:10:03,709 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-71) [6b1f65fb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:10:03,710 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [32006555] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:10:03,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [32006555] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 27e43ef6 >2014-04-29 18:13:03,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [32006555] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:13:03,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [32006555] FINISH, ConnectStorageServerVDSCommand, log id: 27e43ef6 >2014-04-29 18:13:03,718 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [32006555] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:13:03,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 2dd8fe31 >2014-04-29 18:13:04,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2dd8fe31 >2014-04-29 18:13:04,197 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:13:04,210 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-71) [7d8218c1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:13:04,212 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [7d8218c1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 67f48915 >2014-04-29 18:13:04,215 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [7d8218c1] FINISH, SetVdsStatusVDSCommand, log id: 67f48915 >2014-04-29 18:13:04,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [7d8218c1] Correlation ID: 7d8218c1, Job ID: 5a726b4a-9be4-4637-8d19-4cc540d5b0bc, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:13:04,230 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [7d8218c1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:13:04,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [7d8218c1] Correlation ID: 6b1f65fb, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:13:04,236 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-71) [354acd74] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:13:04,237 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-71) [635c6af7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:13:04,238 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [635c6af7] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-92) [1f47c9fd] Autorecovering 1 hosts >2014-04-29 18:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-92) [1f47c9fd] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 663e57a >2014-04-29 18:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] FINISH, SetVdsStatusVDSCommand, log id: 663e57a >2014-04-29 18:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 39c2fd3e >2014-04-29 18:15:00,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 499e0421 >2014-04-29 18:15:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] Failed in GetHardwareInfoVDS method >2014-04-29 18:15:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:15:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@25213849 >2014-04-29 18:15:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] HostName = 10.34.27.209 >2014-04-29 18:15:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:15:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [569f2148] FINISH, GetHardwareInfoVDSCommand, log id: 499e0421 >2014-04-29 18:15:00,737 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [569f2148] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:15:00,796 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-92) [1b99bd4f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:15:00,797 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:15:00,798 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 39c2fd3e >2014-04-29 18:15:00,815 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:15:00,825 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [79c76c3] Correlation ID: 569f2148, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was autorecovered. >2014-04-29 18:15:00,825 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:15:01,606 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [58a99b8b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 46b2aa3e >2014-04-29 18:15:01,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [58a99b8b] Failed in GetHardwareInfoVDS method >2014-04-29 18:15:01,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [58a99b8b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:15:01,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [58a99b8b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7bc6f0b4 >2014-04-29 18:15:01,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [58a99b8b] HostName = 10.34.27.209 >2014-04-29 18:15:01,835 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [58a99b8b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:15:01,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [58a99b8b] FINISH, GetHardwareInfoVDSCommand, log id: 46b2aa3e >2014-04-29 18:15:01,840 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [58a99b8b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:15:01,849 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [58a99b8b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:15:02,071 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-89) [2b624c44] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:15:02,072 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [389c8f34] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:15:02,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [389c8f34] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 178dfa45 >2014-04-29 18:18:02,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [389c8f34] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:18:02,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-89) [389c8f34] FINISH, ConnectStorageServerVDSCommand, log id: 178dfa45 >2014-04-29 18:18:02,080 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-89) [389c8f34] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:18:02,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 76043776 >2014-04-29 18:18:02,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 76043776 >2014-04-29 18:18:02,528 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:18:02,544 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-89) [1310b3d6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:18:02,545 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [1310b3d6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5b5a285e >2014-04-29 18:18:02,548 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [1310b3d6] FINISH, SetVdsStatusVDSCommand, log id: 5b5a285e >2014-04-29 18:18:02,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [1310b3d6] Correlation ID: 1310b3d6, Job ID: c76bc642-db5c-41a6-8dba-c8f32ba0b895, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:18:02,564 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [1310b3d6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:18:02,566 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [1310b3d6] Correlation ID: 2b624c44, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:18:02,570 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [23df502a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:18:02,572 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [1c8b0c67] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:18:02,572 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-89) [1c8b0c67] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-47) [4796b052] Autorecovering 1 hosts >2014-04-29 18:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-47) [4796b052] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2a0c7d85 >2014-04-29 18:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] FINISH, SetVdsStatusVDSCommand, log id: 2a0c7d85 >2014-04-29 18:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3b39531d >2014-04-29 18:20:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7b96f77d >2014-04-29 18:20:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] Failed in GetHardwareInfoVDS method >2014-04-29 18:20:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:20:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@74526eee >2014-04-29 18:20:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] HostName = 10.34.27.209 >2014-04-29 18:20:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:20:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [47d75daf] FINISH, GetHardwareInfoVDSCommand, log id: 7b96f77d >2014-04-29 18:20:00,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [47d75daf] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:20:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-47) [7c625592] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:20:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:20:00,717 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3b39531d >2014-04-29 18:20:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:20:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:20:02,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [6798afa9] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7a2eb1d3 >2014-04-29 18:20:03,172 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [6798afa9] Failed in GetHardwareInfoVDS method >2014-04-29 18:20:03,172 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [6798afa9] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:20:03,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [6798afa9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@f9cd250 >2014-04-29 18:20:03,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [6798afa9] HostName = 10.34.27.209 >2014-04-29 18:20:03,172 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [6798afa9] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:20:03,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [6798afa9] FINISH, GetHardwareInfoVDSCommand, log id: 7a2eb1d3 >2014-04-29 18:20:03,177 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [6798afa9] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:20:03,185 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [6798afa9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:20:03,407 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-95) [4806d0cf] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:20:03,408 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [7f017316] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:20:03,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [7f017316] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 240a0cbd >2014-04-29 18:23:03,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [7f017316] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:23:03,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-95) [7f017316] FINISH, ConnectStorageServerVDSCommand, log id: 240a0cbd >2014-04-29 18:23:03,416 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-95) [7f017316] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:23:03,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 11cbf762 >2014-04-29 18:23:03,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 11cbf762 >2014-04-29 18:23:03,867 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:23:03,881 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-95) [1f82c649] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:23:03,883 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [1f82c649] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6f928a55 >2014-04-29 18:23:03,885 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [1f82c649] FINISH, SetVdsStatusVDSCommand, log id: 6f928a55 >2014-04-29 18:23:03,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [1f82c649] Correlation ID: 1f82c649, Job ID: 828c7e4b-75c0-40bd-9bb9-7ada8d6e44c4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:23:03,901 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [1f82c649] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:23:03,903 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [1f82c649] Correlation ID: 4806d0cf, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:23:03,907 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [167925ae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:23:03,909 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [77f90510] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:23:03,909 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-95) [77f90510] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-30) [67b98f68] Autorecovering 1 hosts >2014-04-29 18:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-30) [67b98f68] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 35046dd2 >2014-04-29 18:25:00,031 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] FINISH, SetVdsStatusVDSCommand, log id: 35046dd2 >2014-04-29 18:25:00,033 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2e225db8 >2014-04-29 18:25:00,502 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 39bd1bde >2014-04-29 18:25:00,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] Failed in GetHardwareInfoVDS method >2014-04-29 18:25:00,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:25:00,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2e70a96f >2014-04-29 18:25:00,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] HostName = 10.34.27.209 >2014-04-29 18:25:00,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:25:00,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [38b3c767] FINISH, GetHardwareInfoVDSCommand, log id: 39bd1bde >2014-04-29 18:25:00,734 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [38b3c767] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:25:00,742 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-30) [3ad03940] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:25:00,743 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:25:00,744 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2e225db8 >2014-04-29 18:25:00,757 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:25:00,757 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:25:01,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 14b03f51 >2014-04-29 18:25:01,486 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] Failed in GetHardwareInfoVDS method >2014-04-29 18:25:01,486 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:25:01,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3f0ce973 >2014-04-29 18:25:01,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] HostName = 10.34.27.209 >2014-04-29 18:25:01,486 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:25:01,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [44b08b53] FINISH, GetHardwareInfoVDSCommand, log id: 14b03f51 >2014-04-29 18:25:01,491 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [44b08b53] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:25:01,503 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [44b08b53] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:25:01,728 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-87) [7d23a063] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:25:01,729 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-87) [52d85817] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:25:01,735 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-87) [52d85817] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 399e24ca >2014-04-29 18:28:01,737 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-87) [52d85817] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:28:01,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-87) [52d85817] FINISH, ConnectStorageServerVDSCommand, log id: 399e24ca >2014-04-29 18:28:01,737 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-87) [52d85817] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:28:01,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3b055a22 >2014-04-29 18:28:02,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3b055a22 >2014-04-29 18:28:02,206 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:28:02,217 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-87) [78364998] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:28:02,218 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [78364998] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5467a0ef >2014-04-29 18:28:02,221 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [78364998] FINISH, SetVdsStatusVDSCommand, log id: 5467a0ef >2014-04-29 18:28:02,227 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [78364998] Correlation ID: 78364998, Job ID: 955d6bdc-2822-4a52-8d7f-06d8a7f32e1e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:28:02,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [78364998] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:28:02,235 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [78364998] Correlation ID: 7d23a063, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:28:02,240 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [59929f98] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:28:02,242 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-87) [116711a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:28:02,242 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-87) [116711a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [4a4eba78] Autorecovering 1 hosts >2014-04-29 18:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [4a4eba78] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7c1d26a4 >2014-04-29 18:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] FINISH, SetVdsStatusVDSCommand, log id: 7c1d26a4 >2014-04-29 18:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3a7152c6 >2014-04-29 18:30:00,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 15dc6866 >2014-04-29 18:30:00,716 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] Failed in GetHardwareInfoVDS method >2014-04-29 18:30:00,716 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:30:00,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@394a0bc0 >2014-04-29 18:30:00,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] HostName = 10.34.27.209 >2014-04-29 18:30:00,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:30:00,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] FINISH, GetHardwareInfoVDSCommand, log id: 15dc6866 >2014-04-29 18:30:00,721 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [18c3c7d4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:30:00,730 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-15) [62214dc7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:30:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-15) [69d0fa4f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:30:00,732 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) [69d0fa4f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3a7152c6 >2014-04-29 18:30:00,745 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [69d0fa4f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:30:00,746 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [69d0fa4f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:30:02,598 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 46edf6bf >2014-04-29 18:30:02,820 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] Failed in GetHardwareInfoVDS method >2014-04-29 18:30:02,820 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:30:02,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@30e299fa >2014-04-29 18:30:02,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] HostName = 10.34.27.209 >2014-04-29 18:30:02,820 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:30:02,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] FINISH, GetHardwareInfoVDSCommand, log id: 46edf6bf >2014-04-29 18:30:02,825 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:30:02,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [54b3c9ce] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:30:03,056 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-60) [3a0572ef] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:30:03,058 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-60) [55bf5201] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:30:03,064 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-60) [55bf5201] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 179d6c8d >2014-04-29 18:33:03,066 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-60) [55bf5201] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:33:03,066 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-60) [55bf5201] FINISH, ConnectStorageServerVDSCommand, log id: 179d6c8d >2014-04-29 18:33:03,066 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-60) [55bf5201] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:33:03,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 55e9598f >2014-04-29 18:33:03,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 55e9598f >2014-04-29 18:33:03,522 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:33:03,533 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-60) [6b59a447] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:33:03,535 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [6b59a447] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5f86a29b >2014-04-29 18:33:03,538 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [6b59a447] FINISH, SetVdsStatusVDSCommand, log id: 5f86a29b >2014-04-29 18:33:03,544 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [6b59a447] Correlation ID: 6b59a447, Job ID: 2c7a9d76-e5f6-4bad-a937-12c9d3cb0893, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:33:03,549 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [6b59a447] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:33:03,551 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [6b59a447] Correlation ID: 3a0572ef, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:33:03,555 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-60) [7ef62b22] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:33:03,556 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-60) [4badcb49] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:33:03,557 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-60) [4badcb49] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-32) [555e6086] Autorecovering 1 hosts >2014-04-29 18:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-32) [555e6086] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1955a9a8 >2014-04-29 18:35:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] FINISH, SetVdsStatusVDSCommand, log id: 1955a9a8 >2014-04-29 18:35:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 54b8c1dd >2014-04-29 18:35:00,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 9e80541 >2014-04-29 18:35:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] Failed in GetHardwareInfoVDS method >2014-04-29 18:35:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:35:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5fb0f37b >2014-04-29 18:35:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] HostName = 10.34.27.209 >2014-04-29 18:35:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:35:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2d547b5c] FINISH, GetHardwareInfoVDSCommand, log id: 9e80541 >2014-04-29 18:35:00,707 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [2d547b5c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:35:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-32) [4c3965df] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:35:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-32) [61d1fd27] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:35:00,718 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) [61d1fd27] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 54b8c1dd >2014-04-29 18:35:00,739 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [61d1fd27] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:35:00,740 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [61d1fd27] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:35:03,909 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [431634a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2af216b8 >2014-04-29 18:35:04,133 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [431634a] Failed in GetHardwareInfoVDS method >2014-04-29 18:35:04,133 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [431634a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:35:04,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [431634a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7bb1a00e >2014-04-29 18:35:04,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [431634a] HostName = 10.34.27.209 >2014-04-29 18:35:04,134 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [431634a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:35:04,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [431634a] FINISH, GetHardwareInfoVDSCommand, log id: 2af216b8 >2014-04-29 18:35:04,139 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [431634a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:35:04,149 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [431634a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:35:04,372 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-33) [291c92e6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:35:04,373 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-33) [4e04a5f1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:35:04,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-33) [4e04a5f1] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 284df857 >2014-04-29 18:36:04,999 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-95) [77f90510] Start refreshing all users data >2014-04-29 18:36:07,267 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) [79c76c3] Attempting to update VMs/Templates Ovf. >2014-04-29 18:36:07,269 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) [79c76c3] Attempting to update VM OVFs in Data Center NFS >2014-04-29 18:36:07,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) [79c76c3] Successfully updated VM OVFs in Data Center NFS >2014-04-29 18:36:07,270 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) [79c76c3] Attempting to update template OVFs in Data Center NFS >2014-04-29 18:36:07,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) [79c76c3] Successfully updated templates OVFs in Data Center NFS >2014-04-29 18:36:07,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) [79c76c3] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 18:36:07,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-92) [79c76c3] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 18:38:04,388 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-33) [4e04a5f1] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:38:04,388 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-33) [4e04a5f1] FINISH, ConnectStorageServerVDSCommand, log id: 284df857 >2014-04-29 18:38:04,388 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-33) [4e04a5f1] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:38:04,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7ac5ce12 >2014-04-29 18:38:04,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7ac5ce12 >2014-04-29 18:38:04,848 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:38:04,993 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-33) [6925efb4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:38:04,995 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [6925efb4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6555ea9c >2014-04-29 18:38:05,022 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [6925efb4] FINISH, SetVdsStatusVDSCommand, log id: 6555ea9c >2014-04-29 18:38:05,052 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [6925efb4] Correlation ID: 6925efb4, Job ID: 2f765b7f-8064-4137-9c27-7b016ba67f5b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:38:05,086 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [6925efb4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:38:05,112 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [6925efb4] Correlation ID: 291c92e6, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:38:05,145 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-33) [382c671d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:38:05,146 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-33) [5f62d99b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:38:05,147 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-33) [5f62d99b] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [76bed1f1] Autorecovering 1 hosts >2014-04-29 18:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [76bed1f1] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:40:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:40:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:40:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:40:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 31ce8e18 >2014-04-29 18:40:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] FINISH, SetVdsStatusVDSCommand, log id: 31ce8e18 >2014-04-29 18:40:00,018 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1346f7a5 >2014-04-29 18:40:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4bfc1680 >2014-04-29 18:40:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] Failed in GetHardwareInfoVDS method >2014-04-29 18:40:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:40:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@14fe290d >2014-04-29 18:40:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] HostName = 10.34.27.209 >2014-04-29 18:40:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:40:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [112da7e2] FINISH, GetHardwareInfoVDSCommand, log id: 4bfc1680 >2014-04-29 18:40:00,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [112da7e2] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:40:00,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-56) [61012cdf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:40:00,730 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-56) [64748dcc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:40:00,731 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [64748dcc] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1346f7a5 >2014-04-29 18:40:00,747 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [64748dcc] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:40:00,749 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [64748dcc] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:40:02,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 78d4d9ab >2014-04-29 18:40:02,737 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] Failed in GetHardwareInfoVDS method >2014-04-29 18:40:02,737 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:40:02,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5e1ea786 >2014-04-29 18:40:02,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] HostName = 10.34.27.209 >2014-04-29 18:40:02,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:40:02,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [2745c35d] FINISH, GetHardwareInfoVDSCommand, log id: 78d4d9ab >2014-04-29 18:40:02,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [2745c35d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:40:02,755 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [2745c35d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:40:02,977 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-65) [1ab3ae84] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:40:02,978 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-65) [343e0c5f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:40:02,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-65) [343e0c5f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 758bd554 >2014-04-29 18:43:02,986 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-65) [343e0c5f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:43:02,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-65) [343e0c5f] FINISH, ConnectStorageServerVDSCommand, log id: 758bd554 >2014-04-29 18:43:02,986 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-65) [343e0c5f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:43:02,989 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5197740f >2014-04-29 18:43:03,445 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 5197740f >2014-04-29 18:43:03,445 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:43:03,456 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-65) [1136cf3a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:43:03,457 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [1136cf3a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3fb77ebb >2014-04-29 18:43:03,459 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [1136cf3a] FINISH, SetVdsStatusVDSCommand, log id: 3fb77ebb >2014-04-29 18:43:03,465 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [1136cf3a] Correlation ID: 1136cf3a, Job ID: fecdd5ac-bd75-41a7-9206-12be3b62b0af, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:43:03,471 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [1136cf3a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:43:03,473 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [1136cf3a] Correlation ID: 1ab3ae84, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:43:03,476 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-65) [3c1146d3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:43:03,477 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-65) [26e34cd9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:43:03,478 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-65) [26e34cd9] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [541ebd1e] Autorecovering 1 hosts >2014-04-29 18:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [541ebd1e] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5f51c82c >2014-04-29 18:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] FINISH, SetVdsStatusVDSCommand, log id: 5f51c82c >2014-04-29 18:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7c021151 >2014-04-29 18:45:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3acf1d6 >2014-04-29 18:45:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] Failed in GetHardwareInfoVDS method >2014-04-29 18:45:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:45:00,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4f391a7c >2014-04-29 18:45:00,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] HostName = 10.34.27.209 >2014-04-29 18:45:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:45:00,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [4c007a9b] FINISH, GetHardwareInfoVDSCommand, log id: 3acf1d6 >2014-04-29 18:45:00,712 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [4c007a9b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:45:00,721 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [76d51fcb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:45:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [2b7c6260] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:45:00,723 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [2b7c6260] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7c021151 >2014-04-29 18:45:00,736 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [2b7c6260] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:45:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [2b7c6260] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:45:03,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 37e0ba5f >2014-04-29 18:45:04,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] Failed in GetHardwareInfoVDS method >2014-04-29 18:45:04,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:45:04,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2ebcfa2 >2014-04-29 18:45:04,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] HostName = 10.34.27.209 >2014-04-29 18:45:04,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:45:04,045 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] FINISH, GetHardwareInfoVDSCommand, log id: 37e0ba5f >2014-04-29 18:45:04,050 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:45:04,059 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [31e7c4e7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:45:04,282 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-12) [4e712777] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:45:04,284 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [6610437] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:45:04,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [6610437] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 39717efa >2014-04-29 18:48:04,298 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [6610437] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:48:04,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [6610437] FINISH, ConnectStorageServerVDSCommand, log id: 39717efa >2014-04-29 18:48:04,299 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [6610437] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:48:04,303 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 69553456 >2014-04-29 18:48:04,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 69553456 >2014-04-29 18:48:04,752 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:48:04,764 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [738c3651] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:48:04,766 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [738c3651] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1db7247e >2014-04-29 18:48:04,768 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [738c3651] FINISH, SetVdsStatusVDSCommand, log id: 1db7247e >2014-04-29 18:48:04,778 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [738c3651] Correlation ID: 738c3651, Job ID: 9bbcb6ac-8af2-4edb-b3b9-a045bdb48a13, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:48:04,785 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [738c3651] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:48:04,787 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [738c3651] Correlation ID: 4e712777, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:48:04,792 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [72d5aba3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:48:04,793 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [58040b92] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:48:04,794 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [58040b92] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-36) [5fb8b447] Autorecovering 1 hosts >2014-04-29 18:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-36) [5fb8b447] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:50:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:50:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:50:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:50:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5fbbd493 >2014-04-29 18:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] FINISH, SetVdsStatusVDSCommand, log id: 5fbbd493 >2014-04-29 18:50:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 177887b8 >2014-04-29 18:50:00,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 188cc896 >2014-04-29 18:50:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] Failed in GetHardwareInfoVDS method >2014-04-29 18:50:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:50:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@42c4fc71 >2014-04-29 18:50:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] HostName = 10.34.27.209 >2014-04-29 18:50:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:50:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [579d0070] FINISH, GetHardwareInfoVDSCommand, log id: 188cc896 >2014-04-29 18:50:00,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [579d0070] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:50:00,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [32a2395a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:50:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [7128c0f5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:50:00,732 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-36) [7128c0f5] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 177887b8 >2014-04-29 18:50:00,751 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [7128c0f5] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:50:00,752 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [7128c0f5] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:50:02,162 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5b51ea41 >2014-04-29 18:50:02,386 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] Failed in GetHardwareInfoVDS method >2014-04-29 18:50:02,386 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:50:02,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5eea803d >2014-04-29 18:50:02,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] HostName = 10.34.27.209 >2014-04-29 18:50:02,386 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:50:02,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [3565555f] FINISH, GetHardwareInfoVDSCommand, log id: 5b51ea41 >2014-04-29 18:50:02,392 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [3565555f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:50:02,401 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [3565555f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:50:02,624 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-14) [998db05] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:50:02,626 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [4802272a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:50:02,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [4802272a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1178b539 >2014-04-29 18:53:02,634 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [4802272a] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:53:02,635 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [4802272a] FINISH, ConnectStorageServerVDSCommand, log id: 1178b539 >2014-04-29 18:53:02,635 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [4802272a] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:53:02,639 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4c432a4e >2014-04-29 18:53:03,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4c432a4e >2014-04-29 18:53:03,079 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:53:03,089 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-14) [6120e630] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:53:03,090 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [6120e630] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 21bf7d2a >2014-04-29 18:53:03,092 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [6120e630] FINISH, SetVdsStatusVDSCommand, log id: 21bf7d2a >2014-04-29 18:53:03,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [6120e630] Correlation ID: 6120e630, Job ID: ca4d9889-e4d4-437b-aa65-4e04919f3e4a, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:53:03,105 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [6120e630] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:53:03,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [6120e630] Correlation ID: 998db05, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:53:03,112 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-14) [4d69faa0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:53:03,113 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-14) [41c42308] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:53:03,114 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [41c42308] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 18:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [69d0fa4f] Autorecovering 1 hosts >2014-04-29 18:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-15) [69d0fa4f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 18:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 18:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 18:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 326ef45 >2014-04-29 18:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] FINISH, SetVdsStatusVDSCommand, log id: 326ef45 >2014-04-29 18:55:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6c994830 >2014-04-29 18:55:00,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: ba535f2 >2014-04-29 18:55:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] Failed in GetHardwareInfoVDS method >2014-04-29 18:55:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:55:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3198002a >2014-04-29 18:55:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] HostName = 10.34.27.209 >2014-04-29 18:55:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:55:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] FINISH, GetHardwareInfoVDSCommand, log id: ba535f2 >2014-04-29 18:55:00,719 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [5ce4e9de] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:55:00,730 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-15) [3823a156] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:55:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:55:00,732 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6c994830 >2014-04-29 18:55:00,745 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 18:55:00,746 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 18:55:03,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4f6dadcb >2014-04-29 18:55:03,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) Failed in GetHardwareInfoVDS method >2014-04-29 18:55:03,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:55:03,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@29ad132a >2014-04-29 18:55:03,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) HostName = 10.34.27.209 >2014-04-29 18:55:03,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 18:55:03,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) FINISH, GetHardwareInfoVDSCommand, log id: 4f6dadcb >2014-04-29 18:55:03,719 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 18:55:03,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 18:55:03,955 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-38) [32acfb41] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:55:03,959 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-38) [25f95bee] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 18:55:03,966 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [25f95bee] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7b66dad1 >2014-04-29 18:58:03,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [25f95bee] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 18:58:03,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [25f95bee] FINISH, ConnectStorageServerVDSCommand, log id: 7b66dad1 >2014-04-29 18:58:03,967 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-38) [25f95bee] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 18:58:03,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 73ca80e9 >2014-04-29 18:58:04,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 73ca80e9 >2014-04-29 18:58:04,411 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 18:58:04,438 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-38) [211c9bff] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:58:04,440 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [211c9bff] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: f89ce28 >2014-04-29 18:58:04,443 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [211c9bff] FINISH, SetVdsStatusVDSCommand, log id: f89ce28 >2014-04-29 18:58:04,450 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [211c9bff] Correlation ID: 211c9bff, Job ID: d8f087b5-38a8-465f-b297-89cd54af709b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 18:58:04,456 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [211c9bff] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 18:58:04,458 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [211c9bff] Correlation ID: 32acfb41, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 18:58:04,462 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-38) [471ab5fb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:58:04,463 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 18:58:04,464 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-17) Autorecovering 1 hosts >2014-04-29 19:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-17) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1725bc96 >2014-04-29 19:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] FINISH, SetVdsStatusVDSCommand, log id: 1725bc96 >2014-04-29 19:00:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2d06cf92 >2014-04-29 19:00:00,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: c3896fe >2014-04-29 19:00:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] Failed in GetHardwareInfoVDS method >2014-04-29 19:00:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:00:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5eb4bfb1 >2014-04-29 19:00:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] HostName = 10.34.27.209 >2014-04-29 19:00:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:00:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [4d575b7c] FINISH, GetHardwareInfoVDSCommand, log id: c3896fe >2014-04-29 19:00:00,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [4d575b7c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:00:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-17) [15530c6c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:00:00,733 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:00:00,734 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2d06cf92 >2014-04-29 19:00:00,753 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:00:00,754 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:00:01,820 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3ce72545 >2014-04-29 19:00:02,042 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] Failed in GetHardwareInfoVDS method >2014-04-29 19:00:02,042 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:00:02,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@770aa398 >2014-04-29 19:00:02,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] HostName = 10.34.27.209 >2014-04-29 19:00:02,042 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:00:02,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] FINISH, GetHardwareInfoVDSCommand, log id: 3ce72545 >2014-04-29 19:00:02,069 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:00:02,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [1eabf4a5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:00:02,352 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-74) [4ed0260b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:00:02,356 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-74) [78a66125] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:00:02,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-74) [78a66125] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 430913ac >2014-04-29 19:03:02,367 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-74) [78a66125] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:03:02,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-74) [78a66125] FINISH, ConnectStorageServerVDSCommand, log id: 430913ac >2014-04-29 19:03:02,367 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-74) [78a66125] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:03:02,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 9e1dc00 >2014-04-29 19:03:02,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 9e1dc00 >2014-04-29 19:03:02,823 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:03:02,837 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-74) [1189b6a9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:03:02,838 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) [1189b6a9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4cf45092 >2014-04-29 19:03:02,841 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-74) [1189b6a9] FINISH, SetVdsStatusVDSCommand, log id: 4cf45092 >2014-04-29 19:03:02,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [1189b6a9] Correlation ID: 1189b6a9, Job ID: cf386741-c291-4b89-86ab-c662fd5b1dc0, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:03:02,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [1189b6a9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:03:02,865 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-74) [1189b6a9] Correlation ID: 4ed0260b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:03:02,869 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-74) [27781fe0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:03:02,871 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-74) [76adfcce] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:03:02,872 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-74) [76adfcce] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-73) [639730bf] Autorecovering 1 hosts >2014-04-29 19:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-73) [639730bf] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 282802e9 >2014-04-29 19:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] FINISH, SetVdsStatusVDSCommand, log id: 282802e9 >2014-04-29 19:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5e12b67 >2014-04-29 19:05:00,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2392297b >2014-04-29 19:05:00,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] Failed in GetHardwareInfoVDS method >2014-04-29 19:05:00,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:05:00,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@50e15c7b >2014-04-29 19:05:00,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] HostName = 10.34.27.209 >2014-04-29 19:05:00,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:05:00,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [478a2299] FINISH, GetHardwareInfoVDSCommand, log id: 2392297b >2014-04-29 19:05:00,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [478a2299] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:05:00,754 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [15669d7b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:05:00,755 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:05:00,756 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5e12b67 >2014-04-29 19:05:00,771 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:05:00,772 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:05:03,298 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 78a0e245 >2014-04-29 19:05:03,520 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] Failed in GetHardwareInfoVDS method >2014-04-29 19:05:03,520 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:05:03,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@a675de2 >2014-04-29 19:05:03,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] HostName = 10.34.27.209 >2014-04-29 19:05:03,521 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:05:03,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [79c76c3] FINISH, GetHardwareInfoVDSCommand, log id: 78a0e245 >2014-04-29 19:05:03,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [79c76c3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:05:03,553 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [79c76c3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:05:03,776 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-92) [5e7da7c1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:05:03,778 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-92) [3e499710] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:05:03,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [3e499710] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 52f6eaa2 >2014-04-29 19:08:03,786 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [3e499710] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:08:03,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [3e499710] FINISH, ConnectStorageServerVDSCommand, log id: 52f6eaa2 >2014-04-29 19:08:03,786 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-92) [3e499710] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:08:03,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4971dead >2014-04-29 19:08:04,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4971dead >2014-04-29 19:08:04,258 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:08:04,268 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-92) [6e2ada84] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:08:04,269 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [6e2ada84] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3e2bffd0 >2014-04-29 19:08:04,271 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [6e2ada84] FINISH, SetVdsStatusVDSCommand, log id: 3e2bffd0 >2014-04-29 19:08:04,278 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [6e2ada84] Correlation ID: 6e2ada84, Job ID: 5fbea41b-eed9-47f5-bd46-0902d1ed9f3e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:08:04,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [6e2ada84] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:08:04,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [6e2ada84] Correlation ID: 5e7da7c1, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:08:04,290 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-92) [31758977] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:08:04,291 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:08:04,292 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [2b7c6260] Autorecovering 1 hosts >2014-04-29 19:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [2b7c6260] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 70acd205 >2014-04-29 19:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] FINISH, SetVdsStatusVDSCommand, log id: 70acd205 >2014-04-29 19:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5ec84bf8 >2014-04-29 19:10:00,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 57b1c3c0 >2014-04-29 19:10:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] Failed in GetHardwareInfoVDS method >2014-04-29 19:10:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:10:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@57aa9f3a >2014-04-29 19:10:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] HostName = 10.34.27.209 >2014-04-29 19:10:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:10:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [1857fb21] FINISH, GetHardwareInfoVDSCommand, log id: 57b1c3c0 >2014-04-29 19:10:00,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [1857fb21] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:10:00,726 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [fcb095f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:10:00,727 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [49295470] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:10:00,728 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [49295470] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5ec84bf8 >2014-04-29 19:10:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [49295470] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:10:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [49295470] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:10:01,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2b39e940 >2014-04-29 19:10:01,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] Failed in GetHardwareInfoVDS method >2014-04-29 19:10:01,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:10:01,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3040ef32 >2014-04-29 19:10:01,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] HostName = 10.34.27.209 >2014-04-29 19:10:01,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:10:01,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [4799dd1e] FINISH, GetHardwareInfoVDSCommand, log id: 2b39e940 >2014-04-29 19:10:01,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [4799dd1e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:10:01,885 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [4799dd1e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:10:02,107 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-85) [553912c9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:10:02,109 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-85) [2882b7cf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:10:02,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-85) [2882b7cf] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 59a9f4fd >2014-04-29 19:13:02,117 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-85) [2882b7cf] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:13:02,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-85) [2882b7cf] FINISH, ConnectStorageServerVDSCommand, log id: 59a9f4fd >2014-04-29 19:13:02,117 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-85) [2882b7cf] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:13:02,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 6d4b4328 >2014-04-29 19:13:02,626 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6d4b4328 >2014-04-29 19:13:02,626 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:13:02,638 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-85) [1264ef7d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:13:02,639 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [1264ef7d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4eae1e9e >2014-04-29 19:13:02,642 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [1264ef7d] FINISH, SetVdsStatusVDSCommand, log id: 4eae1e9e >2014-04-29 19:13:02,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [1264ef7d] Correlation ID: 1264ef7d, Job ID: e22ad7d8-ee3a-48a2-8ce8-59ba6a884fc5, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:13:02,656 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [1264ef7d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:13:02,658 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [1264ef7d] Correlation ID: 553912c9, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:13:02,664 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-85) [2e81fc0a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:13:02,665 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-85) [458a8012] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:13:02,669 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [458a8012] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [7ca93cb0] Autorecovering 1 hosts >2014-04-29 19:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [7ca93cb0] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 521b2a48 >2014-04-29 19:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] FINISH, SetVdsStatusVDSCommand, log id: 521b2a48 >2014-04-29 19:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 79b34e87 >2014-04-29 19:15:00,519 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7e5ce81e >2014-04-29 19:15:00,743 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] Failed in GetHardwareInfoVDS method >2014-04-29 19:15:00,743 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:15:00,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@58b4a62 >2014-04-29 19:15:00,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] HostName = 10.34.27.209 >2014-04-29 19:15:00,744 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:15:00,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [45f910c5] FINISH, GetHardwareInfoVDSCommand, log id: 7e5ce81e >2014-04-29 19:15:00,750 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [45f910c5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:15:00,762 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-97) [49743934] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:15:00,763 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-97) [5b3461c7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:15:00,764 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) [5b3461c7] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 79b34e87 >2014-04-29 19:15:00,781 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [5b3461c7] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:15:00,782 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [5b3461c7] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:15:03,029 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 23b5e8f6 >2014-04-29 19:15:03,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] Failed in GetHardwareInfoVDS method >2014-04-29 19:15:03,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:15:03,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@45e62948 >2014-04-29 19:15:03,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] HostName = 10.34.27.209 >2014-04-29 19:15:03,251 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:15:03,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-40) [769b5849] FINISH, GetHardwareInfoVDSCommand, log id: 23b5e8f6 >2014-04-29 19:15:03,256 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [769b5849] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:15:03,265 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [769b5849] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:15:03,487 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-40) [23f348e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:15:03,489 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-40) [4f5c2b40] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:15:03,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-40) [4f5c2b40] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: e936338 >2014-04-29 19:18:03,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-40) [4f5c2b40] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:18:03,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-40) [4f5c2b40] FINISH, ConnectStorageServerVDSCommand, log id: e936338 >2014-04-29 19:18:03,497 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-40) [4f5c2b40] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:18:03,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7e3b5865 >2014-04-29 19:18:03,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 7e3b5865 >2014-04-29 19:18:03,956 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:18:03,968 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-40) [192fac8a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:18:03,969 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [192fac8a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3da11ef7 >2014-04-29 19:18:03,971 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-40) [192fac8a] FINISH, SetVdsStatusVDSCommand, log id: 3da11ef7 >2014-04-29 19:18:03,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [192fac8a] Correlation ID: 192fac8a, Job ID: f9d3f8d3-8127-40b2-9226-f5da1d67e094, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:18:03,983 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [192fac8a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:18:03,985 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-40) [192fac8a] Correlation ID: 23f348e, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:18:03,991 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-40) [4badfd06] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:18:03,992 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-40) [765bacdb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:18:03,993 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-40) [765bacdb] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-5) [21573f3a] Autorecovering 1 hosts >2014-04-29 19:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-5) [21573f3a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: f4f19e2 >2014-04-29 19:20:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] FINISH, SetVdsStatusVDSCommand, log id: f4f19e2 >2014-04-29 19:20:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 84978e7 >2014-04-29 19:20:00,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 297087f2 >2014-04-29 19:20:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] Failed in GetHardwareInfoVDS method >2014-04-29 19:20:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:20:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1aed361f >2014-04-29 19:20:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] HostName = 10.34.27.209 >2014-04-29 19:20:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:20:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [5ca06243] FINISH, GetHardwareInfoVDSCommand, log id: 297087f2 >2014-04-29 19:20:00,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [5ca06243] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:20:00,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-5) [67fa95a4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:20:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-5) [72c4daef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:20:00,731 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) [72c4daef] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 84978e7 >2014-04-29 19:20:00,747 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [72c4daef] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:20:00,749 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [72c4daef] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:20:01,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1fa8c9fa >2014-04-29 19:20:01,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] Failed in GetHardwareInfoVDS method >2014-04-29 19:20:01,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:20:01,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2644c53 >2014-04-29 19:20:01,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] HostName = 10.34.27.209 >2014-04-29 19:20:01,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:20:01,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [8f60cef] FINISH, GetHardwareInfoVDSCommand, log id: 1fa8c9fa >2014-04-29 19:20:01,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [8f60cef] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:20:01,619 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [8f60cef] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:20:01,847 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-15) [11970769] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:20:01,848 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-15) [396b5678] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:20:01,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [396b5678] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 552e3ff0 >2014-04-29 19:23:01,856 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [396b5678] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:23:01,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [396b5678] FINISH, ConnectStorageServerVDSCommand, log id: 552e3ff0 >2014-04-29 19:23:01,856 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-15) [396b5678] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:23:01,859 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 2b757761 >2014-04-29 19:23:02,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2b757761 >2014-04-29 19:23:02,299 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:23:02,312 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-15) [3f25b5ef] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:23:02,313 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [3f25b5ef] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4681b95d >2014-04-29 19:23:02,316 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [3f25b5ef] FINISH, SetVdsStatusVDSCommand, log id: 4681b95d >2014-04-29 19:23:02,327 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [3f25b5ef] Correlation ID: 3f25b5ef, Job ID: 4a9b88a6-8892-4741-93a4-c92ed3bb215d, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:23:02,338 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [3f25b5ef] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:23:02,340 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [3f25b5ef] Correlation ID: 11970769, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:23:02,345 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-15) [4a5fd8d2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:23:02,346 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-15) [347a0b4d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:23:02,348 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [347a0b4d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [44666754] Autorecovering 1 hosts >2014-04-29 19:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [44666754] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4197994f >2014-04-29 19:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] FINISH, SetVdsStatusVDSCommand, log id: 4197994f >2014-04-29 19:25:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 45e955bb >2014-04-29 19:25:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1877651 >2014-04-29 19:25:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] Failed in GetHardwareInfoVDS method >2014-04-29 19:25:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:25:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6a074720 >2014-04-29 19:25:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] HostName = 10.34.27.209 >2014-04-29 19:25:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:25:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5e529e4e] FINISH, GetHardwareInfoVDSCommand, log id: 1877651 >2014-04-29 19:25:00,710 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [5e529e4e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:25:00,718 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [7369dc45] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:25:00,719 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:25:00,720 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 45e955bb >2014-04-29 19:25:00,736 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:25:00,736 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:25:02,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7078fffe >2014-04-29 19:25:02,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] Failed in GetHardwareInfoVDS method >2014-04-29 19:25:02,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:25:02,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@17eeb662 >2014-04-29 19:25:02,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] HostName = 10.34.27.209 >2014-04-29 19:25:02,941 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:25:02,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] FINISH, GetHardwareInfoVDSCommand, log id: 7078fffe >2014-04-29 19:25:02,946 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:25:02,955 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [2e58e4cf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:25:03,176 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-17) [2a4b02e9] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:25:03,180 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-17) [51720cdb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:25:03,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-17) [51720cdb] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1cedaeb0 >2014-04-29 19:28:03,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-17) [51720cdb] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:28:03,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-17) [51720cdb] FINISH, ConnectStorageServerVDSCommand, log id: 1cedaeb0 >2014-04-29 19:28:03,189 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-17) [51720cdb] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:28:03,196 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7c3dded7 >2014-04-29 19:28:03,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 7c3dded7 >2014-04-29 19:28:03,637 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:28:03,679 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-17) [4597678b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:28:03,680 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [4597678b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 38b8d139 >2014-04-29 19:28:03,683 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [4597678b] FINISH, SetVdsStatusVDSCommand, log id: 38b8d139 >2014-04-29 19:28:03,690 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [4597678b] Correlation ID: 4597678b, Job ID: 9452c7c7-c13c-4f96-a846-530cc00fefb5, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:28:03,697 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [4597678b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:28:03,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [4597678b] Correlation ID: 2a4b02e9, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:28:03,703 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-17) [1450876f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:28:03,708 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-17) [c5ba562] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:28:03,709 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) [c5ba562] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-53) [55817744] Autorecovering 1 hosts >2014-04-29 19:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-53) [55817744] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 70e35eed >2014-04-29 19:30:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] FINISH, SetVdsStatusVDSCommand, log id: 70e35eed >2014-04-29 19:30:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 66374aa6 >2014-04-29 19:30:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5c9e187f >2014-04-29 19:30:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] Failed in GetHardwareInfoVDS method >2014-04-29 19:30:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:30:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@f5a695a >2014-04-29 19:30:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] HostName = 10.34.27.209 >2014-04-29 19:30:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:30:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [79206a65] FINISH, GetHardwareInfoVDSCommand, log id: 5c9e187f >2014-04-29 19:30:00,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [79206a65] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:30:00,733 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-53) [28b0ec54] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:30:00,734 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-53) [2a286eea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:30:00,735 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) [2a286eea] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 66374aa6 >2014-04-29 19:30:00,765 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [2a286eea] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:30:00,766 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [2a286eea] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:30:01,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [15eddc0c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 705e0a18 >2014-04-29 19:30:01,284 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [15eddc0c] Failed in GetHardwareInfoVDS method >2014-04-29 19:30:01,284 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [15eddc0c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:30:01,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [15eddc0c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2de8e74a >2014-04-29 19:30:01,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [15eddc0c] HostName = 10.34.27.209 >2014-04-29 19:30:01,284 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [15eddc0c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:30:01,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [15eddc0c] FINISH, GetHardwareInfoVDSCommand, log id: 705e0a18 >2014-04-29 19:30:01,289 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [15eddc0c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:30:01,300 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [15eddc0c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:30:01,522 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-29) [1da25033] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:30:01,527 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [4dd3da33] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:30:01,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [4dd3da33] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7e8c1c3f >2014-04-29 19:33:01,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [4dd3da33] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:33:01,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [4dd3da33] FINISH, ConnectStorageServerVDSCommand, log id: 7e8c1c3f >2014-04-29 19:33:01,537 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [4dd3da33] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:33:01,541 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1bc9cc62 >2014-04-29 19:33:02,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 1bc9cc62 >2014-04-29 19:33:02,057 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:33:02,068 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-29) [19a538fa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:33:02,070 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [19a538fa] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 39a2129b >2014-04-29 19:33:02,072 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [19a538fa] FINISH, SetVdsStatusVDSCommand, log id: 39a2129b >2014-04-29 19:33:02,078 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [19a538fa] Correlation ID: 19a538fa, Job ID: 2380f784-ebdc-40b5-af4c-cd9d13240e4b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:33:02,092 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [19a538fa] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:33:02,093 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [19a538fa] Correlation ID: 1da25033, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:33:02,098 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-29) [f7c1f5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:33:02,099 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-29) [5c65b29f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:33:02,100 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) [5c65b29f] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) [5628a0bc] Autorecovering 1 hosts >2014-04-29 19:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) [5628a0bc] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 11f154fb >2014-04-29 19:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] FINISH, SetVdsStatusVDSCommand, log id: 11f154fb >2014-04-29 19:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 597bd64f >2014-04-29 19:35:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 28db82b2 >2014-04-29 19:35:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] Failed in GetHardwareInfoVDS method >2014-04-29 19:35:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:35:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4719429f >2014-04-29 19:35:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] HostName = 10.34.27.209 >2014-04-29 19:35:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:35:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [de5ef8f] FINISH, GetHardwareInfoVDSCommand, log id: 28db82b2 >2014-04-29 19:35:00,710 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [de5ef8f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:35:00,718 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [4b9c000] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:35:00,719 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [3da45a1c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:35:00,720 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [3da45a1c] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 597bd64f >2014-04-29 19:35:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [3da45a1c] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:35:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [3da45a1c] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:35:02,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [7a93dbee] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 57bfe771 >2014-04-29 19:35:02,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [7a93dbee] Failed in GetHardwareInfoVDS method >2014-04-29 19:35:02,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [7a93dbee] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:35:02,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [7a93dbee] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@77f3d8b8 >2014-04-29 19:35:02,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [7a93dbee] HostName = 10.34.27.209 >2014-04-29 19:35:02,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [7a93dbee] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:35:02,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [7a93dbee] FINISH, GetHardwareInfoVDSCommand, log id: 57bfe771 >2014-04-29 19:35:02,701 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [7a93dbee] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:35:02,710 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [7a93dbee] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:35:02,933 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-31) [39c2353f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:35:02,935 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [7c587bb1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:35:02,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [7c587bb1] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 412990c8 >2014-04-29 19:36:05,002 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-51) [535824f4] Start refreshing all users data >2014-04-29 19:36:07,271 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) [51c90391] Attempting to update VMs/Templates Ovf. >2014-04-29 19:36:07,273 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) [51c90391] Attempting to update VM OVFs in Data Center NFS >2014-04-29 19:36:07,273 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) [51c90391] Successfully updated VM OVFs in Data Center NFS >2014-04-29 19:36:07,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) [51c90391] Attempting to update template OVFs in Data Center NFS >2014-04-29 19:36:07,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) [51c90391] Successfully updated templates OVFs in Data Center NFS >2014-04-29 19:36:07,274 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) [51c90391] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 19:36:07,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-16) [51c90391] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 19:38:02,943 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [7c587bb1] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:38:02,943 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [7c587bb1] FINISH, ConnectStorageServerVDSCommand, log id: 412990c8 >2014-04-29 19:38:02,943 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [7c587bb1] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:38:02,946 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 6e77526c >2014-04-29 19:38:03,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6e77526c >2014-04-29 19:38:03,418 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:38:03,430 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-31) [2876e64b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:38:03,433 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [2876e64b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 30876332 >2014-04-29 19:38:03,436 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [2876e64b] FINISH, SetVdsStatusVDSCommand, log id: 30876332 >2014-04-29 19:38:03,450 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [2876e64b] Correlation ID: 2876e64b, Job ID: 8284ea0d-fa19-4084-92f6-8ec048ef1fa6, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:38:03,468 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [2876e64b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:38:03,474 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [2876e64b] Correlation ID: 39c2353f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:38:03,483 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-31) [3ed7fa72] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:38:03,485 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-31) [72992f90] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:38:03,486 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [72992f90] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-94) [181e5b1e] Autorecovering 1 hosts >2014-04-29 19:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-94) [181e5b1e] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 384d3b5f >2014-04-29 19:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] FINISH, SetVdsStatusVDSCommand, log id: 384d3b5f >2014-04-29 19:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: aa4499e >2014-04-29 19:40:00,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 72ea4aea >2014-04-29 19:40:00,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] Failed in GetHardwareInfoVDS method >2014-04-29 19:40:00,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:40:00,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@73dc8396 >2014-04-29 19:40:00,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] HostName = 10.34.27.209 >2014-04-29 19:40:00,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:40:00,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [5896c688] FINISH, GetHardwareInfoVDSCommand, log id: 72ea4aea >2014-04-29 19:40:00,733 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [5896c688] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:40:00,753 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-94) [7a9e3d72] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:40:00,754 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-94) [6ae8ff6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:40:00,755 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) [6ae8ff6] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: aa4499e >2014-04-29 19:40:00,769 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [6ae8ff6] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:40:00,772 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [6ae8ff6] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:40:03,843 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5d164a37 >2014-04-29 19:40:04,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] Failed in GetHardwareInfoVDS method >2014-04-29 19:40:04,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:40:04,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6262909f >2014-04-29 19:40:04,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] HostName = 10.34.27.209 >2014-04-29 19:40:04,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:40:04,068 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [52e1a50b] FINISH, GetHardwareInfoVDSCommand, log id: 5d164a37 >2014-04-29 19:40:04,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [52e1a50b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:40:04,087 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [52e1a50b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:40:04,310 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-100) [500278f8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:40:04,311 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-100) [11cb384e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:40:04,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [11cb384e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6b9ae2c7 >2014-04-29 19:43:04,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [11cb384e] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:43:04,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-100) [11cb384e] FINISH, ConnectStorageServerVDSCommand, log id: 6b9ae2c7 >2014-04-29 19:43:04,320 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-100) [11cb384e] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:43:04,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 27a13b02 >2014-04-29 19:43:04,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 27a13b02 >2014-04-29 19:43:04,772 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:43:04,786 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-100) [4dcee2d2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:43:04,787 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [4dcee2d2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4dc3bbe3 >2014-04-29 19:43:04,789 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [4dcee2d2] FINISH, SetVdsStatusVDSCommand, log id: 4dc3bbe3 >2014-04-29 19:43:04,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [4dcee2d2] Correlation ID: 4dcee2d2, Job ID: 0f923705-5764-4f05-a4d1-308ccb722618, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:43:04,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [4dcee2d2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:43:04,803 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [4dcee2d2] Correlation ID: 500278f8, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:43:04,808 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-100) [6e118b9e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:43:04,809 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-100) [169df275] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:43:04,812 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-100) [169df275] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-91) [196fb384] Autorecovering 1 hosts >2014-04-29 19:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-91) [196fb384] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 21991866 >2014-04-29 19:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] FINISH, SetVdsStatusVDSCommand, log id: 21991866 >2014-04-29 19:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2a2d50a5 >2014-04-29 19:45:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 624a4e9 >2014-04-29 19:45:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] Failed in GetHardwareInfoVDS method >2014-04-29 19:45:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:45:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4fc6c025 >2014-04-29 19:45:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] HostName = 10.34.27.209 >2014-04-29 19:45:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:45:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [283dfb50] FINISH, GetHardwareInfoVDSCommand, log id: 624a4e9 >2014-04-29 19:45:00,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [283dfb50] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:45:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [30af8b75] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:45:00,723 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:45:00,724 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2a2d50a5 >2014-04-29 19:45:00,745 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:45:00,747 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:45:02,172 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [58ce83cd] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7246b1c3 >2014-04-29 19:45:02,394 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [58ce83cd] Failed in GetHardwareInfoVDS method >2014-04-29 19:45:02,394 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [58ce83cd] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:45:02,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [58ce83cd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7f40b3d1 >2014-04-29 19:45:02,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [58ce83cd] HostName = 10.34.27.209 >2014-04-29 19:45:02,394 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [58ce83cd] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:45:02,394 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [58ce83cd] FINISH, GetHardwareInfoVDSCommand, log id: 7246b1c3 >2014-04-29 19:45:02,401 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [58ce83cd] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:45:02,409 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [58ce83cd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:45:02,632 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-72) [6b39ac2a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:45:02,637 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [5405eb6b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:45:02,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [5405eb6b] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 398aa675 >2014-04-29 19:48:02,645 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [5405eb6b] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:48:02,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [5405eb6b] FINISH, ConnectStorageServerVDSCommand, log id: 398aa675 >2014-04-29 19:48:02,646 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [5405eb6b] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:48:02,652 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 67fc4f79 >2014-04-29 19:48:03,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 67fc4f79 >2014-04-29 19:48:03,095 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:48:03,107 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-72) [412e5194] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:48:03,108 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [412e5194] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 15db75c4 >2014-04-29 19:48:03,111 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [412e5194] FINISH, SetVdsStatusVDSCommand, log id: 15db75c4 >2014-04-29 19:48:03,118 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [412e5194] Correlation ID: 412e5194, Job ID: ddcb96c3-ca31-4312-8c9f-211c6a4f1106, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:48:03,124 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [412e5194] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:48:03,126 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [412e5194] Correlation ID: 6b39ac2a, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:48:03,130 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-72) [118bcad0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:48:03,131 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-72) [7b8d2f5d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:48:03,131 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [7b8d2f5d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-21) [5c279649] Autorecovering 1 hosts >2014-04-29 19:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-21) [5c279649] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 686e85a1 >2014-04-29 19:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] FINISH, SetVdsStatusVDSCommand, log id: 686e85a1 >2014-04-29 19:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 771546c6 >2014-04-29 19:50:00,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7f56c046 >2014-04-29 19:50:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] Failed in GetHardwareInfoVDS method >2014-04-29 19:50:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:50:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@186974fa >2014-04-29 19:50:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] HostName = 10.34.27.209 >2014-04-29 19:50:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:50:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] FINISH, GetHardwareInfoVDSCommand, log id: 7f56c046 >2014-04-29 19:50:00,725 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [1f6fa6d8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:50:00,734 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-21) [b9bfec] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:50:00,735 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:50:00,736 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 771546c6 >2014-04-29 19:50:00,749 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:50:00,750 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:50:03,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5419206a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: bd43ac3 >2014-04-29 19:50:03,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5419206a] Failed in GetHardwareInfoVDS method >2014-04-29 19:50:03,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5419206a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:50:03,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5419206a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@49679371 >2014-04-29 19:50:03,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5419206a] HostName = 10.34.27.209 >2014-04-29 19:50:03,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5419206a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:50:03,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5419206a] FINISH, GetHardwareInfoVDSCommand, log id: bd43ac3 >2014-04-29 19:50:03,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [5419206a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:50:03,724 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [5419206a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:50:03,949 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-34) [4608a2bf] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:50:03,950 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [c95b547] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:50:03,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [c95b547] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4e1bf11f >2014-04-29 19:53:03,959 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [c95b547] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:53:03,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-34) [c95b547] FINISH, ConnectStorageServerVDSCommand, log id: 4e1bf11f >2014-04-29 19:53:03,959 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-34) [c95b547] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:53:03,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 589baae8 >2014-04-29 19:53:04,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 589baae8 >2014-04-29 19:53:04,414 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:53:04,425 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-34) [43d27590] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:53:04,426 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [43d27590] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 54113a8d >2014-04-29 19:53:04,428 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [43d27590] FINISH, SetVdsStatusVDSCommand, log id: 54113a8d >2014-04-29 19:53:04,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [43d27590] Correlation ID: 43d27590, Job ID: 03a230d4-3a4e-4004-a728-83c1e37cd60c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:53:04,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [43d27590] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:53:04,443 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [43d27590] Correlation ID: 4608a2bf, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:53:04,447 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [13143f79] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:53:04,448 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [67993f0e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:53:04,449 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-34) [67993f0e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 19:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [17b82c51] Autorecovering 1 hosts >2014-04-29 19:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [17b82c51] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 19:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 19:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 19:55:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4588ba15 >2014-04-29 19:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] FINISH, SetVdsStatusVDSCommand, log id: 4588ba15 >2014-04-29 19:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 467a82d5 >2014-04-29 19:55:00,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 34c17da1 >2014-04-29 19:55:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] Failed in GetHardwareInfoVDS method >2014-04-29 19:55:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:55:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@b1ef016 >2014-04-29 19:55:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] HostName = 10.34.27.209 >2014-04-29 19:55:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:55:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [77aa4a50] FINISH, GetHardwareInfoVDSCommand, log id: 34c17da1 >2014-04-29 19:55:00,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [77aa4a50] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:55:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [4a5f8783] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:55:00,718 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:55:00,718 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 467a82d5 >2014-04-29 19:55:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 19:55:00,734 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 19:55:01,815 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 69981ae >2014-04-29 19:55:02,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] Failed in GetHardwareInfoVDS method >2014-04-29 19:55:02,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:55:02,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4810c9e4 >2014-04-29 19:55:02,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] HostName = 10.34.27.209 >2014-04-29 19:55:02,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 19:55:02,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] FINISH, GetHardwareInfoVDSCommand, log id: 69981ae >2014-04-29 19:55:02,043 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 19:55:02,053 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [1df6d7cf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 19:55:02,281 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-41) [4f64a9f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:55:02,283 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-41) [7c89eacf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 19:55:02,289 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-41) [7c89eacf] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 372aa974 >2014-04-29 19:58:02,291 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-41) [7c89eacf] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 19:58:02,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-41) [7c89eacf] FINISH, ConnectStorageServerVDSCommand, log id: 372aa974 >2014-04-29 19:58:02,291 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-41) [7c89eacf] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 19:58:02,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 30534078 >2014-04-29 19:58:02,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 30534078 >2014-04-29 19:58:02,734 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 19:58:02,746 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-41) [3c7eaa6f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:58:02,748 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [3c7eaa6f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 10949209 >2014-04-29 19:58:02,750 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [3c7eaa6f] FINISH, SetVdsStatusVDSCommand, log id: 10949209 >2014-04-29 19:58:02,758 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [3c7eaa6f] Correlation ID: 3c7eaa6f, Job ID: b51f2484-5353-45c8-b8d9-0bb98f9f1b5b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 19:58:02,765 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [3c7eaa6f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 19:58:02,767 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [3c7eaa6f] Correlation ID: 4f64a9f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 19:58:02,771 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-41) [29d35a69] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:58:02,772 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-41) [e55af09] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 19:58:02,773 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) [e55af09] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Autorecovering 1 hosts >2014-04-29 20:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-88) [35b64ebe] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: ea61a58 >2014-04-29 20:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] FINISH, SetVdsStatusVDSCommand, log id: ea61a58 >2014-04-29 20:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 56707e45 >2014-04-29 20:00:00,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4dd843b0 >2014-04-29 20:00:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] Failed in GetHardwareInfoVDS method >2014-04-29 20:00:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:00:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@383cf117 >2014-04-29 20:00:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] HostName = 10.34.27.209 >2014-04-29 20:00:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:00:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [15671b47] FINISH, GetHardwareInfoVDSCommand, log id: 4dd843b0 >2014-04-29 20:00:00,727 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [15671b47] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:00:00,787 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-88) [3d75bc59] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:00:00,789 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:00:00,789 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 56707e45 >2014-04-29 20:00:00,822 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:00:00,823 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:00:03,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6c35c14c >2014-04-29 20:00:03,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] Failed in GetHardwareInfoVDS method >2014-04-29 20:00:03,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:00:03,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@76841f28 >2014-04-29 20:00:03,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] HostName = 10.34.27.209 >2014-04-29 20:00:03,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:00:03,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [516b5f99] FINISH, GetHardwareInfoVDSCommand, log id: 6c35c14c >2014-04-29 20:00:03,362 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [516b5f99] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:00:03,370 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [516b5f99] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:00:03,591 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-47) [59d6f653] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:00:03,593 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-47) [4fdada3b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:00:03,599 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-47) [4fdada3b] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 75ac59b8 >2014-04-29 20:03:03,601 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-47) [4fdada3b] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:03:03,601 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-47) [4fdada3b] FINISH, ConnectStorageServerVDSCommand, log id: 75ac59b8 >2014-04-29 20:03:03,601 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-47) [4fdada3b] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:03:03,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 139f0d7 >2014-04-29 20:03:04,048 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 139f0d7 >2014-04-29 20:03:04,049 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:03:04,058 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-47) [695e9473] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:03:04,059 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [695e9473] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 706acf96 >2014-04-29 20:03:04,062 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [695e9473] FINISH, SetVdsStatusVDSCommand, log id: 706acf96 >2014-04-29 20:03:04,067 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [695e9473] Correlation ID: 695e9473, Job ID: 853f719a-33b6-447a-9488-827d4b87ae94, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:03:04,074 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [695e9473] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:03:04,076 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [695e9473] Correlation ID: 59d6f653, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:03:04,080 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-47) [aaf1888] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:03:04,081 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-47) [22eeda0a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:03:04,082 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [22eeda0a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-16) [51c90391] Autorecovering 1 hosts >2014-04-29 20:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-16) [51c90391] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2b7244e3 >2014-04-29 20:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] FINISH, SetVdsStatusVDSCommand, log id: 2b7244e3 >2014-04-29 20:05:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 427b6fa1 >2014-04-29 20:05:00,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5df91a46 >2014-04-29 20:05:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] Failed in GetHardwareInfoVDS method >2014-04-29 20:05:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:05:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@d2c19dd >2014-04-29 20:05:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] HostName = 10.34.27.209 >2014-04-29 20:05:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:05:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [725949dc] FINISH, GetHardwareInfoVDSCommand, log id: 5df91a46 >2014-04-29 20:05:00,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [725949dc] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:05:00,720 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-16) [67e106ef] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:05:00,721 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-16) [53de3c6d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:05:00,721 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) [53de3c6d] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 427b6fa1 >2014-04-29 20:05:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [53de3c6d] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:05:00,743 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [53de3c6d] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:05:01,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [1584fc64] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 60e6c729 >2014-04-29 20:05:01,665 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [1584fc64] Failed in GetHardwareInfoVDS method >2014-04-29 20:05:01,665 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [1584fc64] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:05:01,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [1584fc64] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4f1d8672 >2014-04-29 20:05:01,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [1584fc64] HostName = 10.34.27.209 >2014-04-29 20:05:01,666 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [1584fc64] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:05:01,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [1584fc64] FINISH, GetHardwareInfoVDSCommand, log id: 60e6c729 >2014-04-29 20:05:01,671 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [1584fc64] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:05:01,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [1584fc64] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:05:01,904 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-26) [15a2173b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:05:01,906 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [d1dd58] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:05:01,912 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [d1dd58] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3bb91c2a >2014-04-29 20:08:01,914 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [d1dd58] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:08:01,914 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [d1dd58] FINISH, ConnectStorageServerVDSCommand, log id: 3bb91c2a >2014-04-29 20:08:01,914 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [d1dd58] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:08:01,917 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 2f29d333 >2014-04-29 20:08:02,373 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2f29d333 >2014-04-29 20:08:02,374 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:08:02,386 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-26) [2973b3b1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:08:02,387 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [2973b3b1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3ec9f465 >2014-04-29 20:08:02,390 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [2973b3b1] FINISH, SetVdsStatusVDSCommand, log id: 3ec9f465 >2014-04-29 20:08:02,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [2973b3b1] Correlation ID: 2973b3b1, Job ID: e142aa4b-85b8-4c0a-abd9-31bc97c9e901, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:08:02,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [2973b3b1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:08:02,406 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [2973b3b1] Correlation ID: 15a2173b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:08:02,411 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-26) [55a57a83] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:08:02,417 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-26) [42cb5d24] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:08:02,417 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [42cb5d24] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [3e5346e2] Autorecovering 1 hosts >2014-04-29 20:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [3e5346e2] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7e7aa7a8 >2014-04-29 20:10:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] FINISH, SetVdsStatusVDSCommand, log id: 7e7aa7a8 >2014-04-29 20:10:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6b5b26cd >2014-04-29 20:10:00,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 18f29a39 >2014-04-29 20:10:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] Failed in GetHardwareInfoVDS method >2014-04-29 20:10:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:10:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5f6b20fa >2014-04-29 20:10:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] HostName = 10.34.27.209 >2014-04-29 20:10:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:10:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [3f9db091] FINISH, GetHardwareInfoVDSCommand, log id: 18f29a39 >2014-04-29 20:10:00,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [3f9db091] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:10:00,725 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [2e374743] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:10:00,726 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:10:00,727 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6b5b26cd >2014-04-29 20:10:00,741 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:10:00,743 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:10:02,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [347a0b4d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7a46e2d5 >2014-04-29 20:10:03,000 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [347a0b4d] Failed in GetHardwareInfoVDS method >2014-04-29 20:10:03,000 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [347a0b4d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:10:03,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [347a0b4d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@43d64caa >2014-04-29 20:10:03,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [347a0b4d] HostName = 10.34.27.209 >2014-04-29 20:10:03,001 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [347a0b4d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:10:03,001 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [347a0b4d] FINISH, GetHardwareInfoVDSCommand, log id: 7a46e2d5 >2014-04-29 20:10:03,005 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [347a0b4d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:10:03,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [347a0b4d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:10:03,236 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-15) [23815c28] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:10:03,237 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-15) [74b78cbe] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:10:03,244 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [74b78cbe] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 62388117 >2014-04-29 20:13:03,245 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [74b78cbe] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:13:03,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [74b78cbe] FINISH, ConnectStorageServerVDSCommand, log id: 62388117 >2014-04-29 20:13:03,246 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-15) [74b78cbe] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:13:03,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 47972062 >2014-04-29 20:13:03,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 47972062 >2014-04-29 20:13:03,699 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:13:03,710 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-15) [29f76088] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:13:03,711 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [29f76088] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7d8765f8 >2014-04-29 20:13:03,713 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [29f76088] FINISH, SetVdsStatusVDSCommand, log id: 7d8765f8 >2014-04-29 20:13:03,719 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [29f76088] Correlation ID: 29f76088, Job ID: 5e62e02c-0aae-4760-8d7f-7a95c8cd3c92, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:13:03,725 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [29f76088] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:13:03,728 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [29f76088] Correlation ID: 23815c28, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:13:03,734 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-15) [61d9a9db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:13:03,735 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-15) [246429fc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:13:03,736 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [246429fc] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-76) [4c2357c9] Autorecovering 1 hosts >2014-04-29 20:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-76) [4c2357c9] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 26ffbe49 >2014-04-29 20:15:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] FINISH, SetVdsStatusVDSCommand, log id: 26ffbe49 >2014-04-29 20:15:00,018 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 341d1bd8 >2014-04-29 20:15:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: a5d200a >2014-04-29 20:15:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] Failed in GetHardwareInfoVDS method >2014-04-29 20:15:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:15:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@67e275d6 >2014-04-29 20:15:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] HostName = 10.34.27.209 >2014-04-29 20:15:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:15:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [74e744ea] FINISH, GetHardwareInfoVDSCommand, log id: a5d200a >2014-04-29 20:15:00,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [74e744ea] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:15:00,723 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-76) [641ff405] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:15:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-76) [44235e17] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:15:00,725 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [44235e17] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 341d1bd8 >2014-04-29 20:15:00,751 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [44235e17] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:15:00,752 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [44235e17] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:15:01,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 27a9b327 >2014-04-29 20:15:01,317 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] Failed in GetHardwareInfoVDS method >2014-04-29 20:15:01,317 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:15:01,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@487c1cc3 >2014-04-29 20:15:01,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] HostName = 10.34.27.209 >2014-04-29 20:15:01,317 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:15:01,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [3da5ae16] FINISH, GetHardwareInfoVDSCommand, log id: 27a9b327 >2014-04-29 20:15:01,321 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [3da5ae16] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:15:01,329 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [3da5ae16] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:15:01,553 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-59) [7a3c355a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:15:01,554 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-59) [34e71133] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:15:01,561 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [34e71133] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70e39429 >2014-04-29 20:18:01,563 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [34e71133] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:18:01,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [34e71133] FINISH, ConnectStorageServerVDSCommand, log id: 70e39429 >2014-04-29 20:18:01,563 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-59) [34e71133] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:18:01,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7ff48e8f >2014-04-29 20:18:02,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 7ff48e8f >2014-04-29 20:18:02,020 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:18:02,030 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-59) [7e35c20d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:18:02,032 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [7e35c20d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 31912681 >2014-04-29 20:18:02,034 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [7e35c20d] FINISH, SetVdsStatusVDSCommand, log id: 31912681 >2014-04-29 20:18:02,040 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [7e35c20d] Correlation ID: 7e35c20d, Job ID: 7519b990-9894-4a67-aa59-f7a39293f91c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:18:02,046 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [7e35c20d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:18:02,047 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [7e35c20d] Correlation ID: 7a3c355a, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:18:02,051 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-59) [51487ad9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:18:02,052 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-59) [665b237b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:18:02,052 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) [665b237b] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:20:00,010 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-43) [137577be] Autorecovering 1 hosts >2014-04-29 20:20:00,010 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-43) [137577be] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:20:00,012 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:20:00,012 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:20:00,012 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:20:00,016 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:20:00,019 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1b2efc28 >2014-04-29 20:20:00,025 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] FINISH, SetVdsStatusVDSCommand, log id: 1b2efc28 >2014-04-29 20:20:00,026 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 241afb92 >2014-04-29 20:20:00,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 63a9ee2b >2014-04-29 20:20:00,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] Failed in GetHardwareInfoVDS method >2014-04-29 20:20:00,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:20:00,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4591cfd >2014-04-29 20:20:00,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] HostName = 10.34.27.209 >2014-04-29 20:20:00,752 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:20:00,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [31ba4f] FINISH, GetHardwareInfoVDSCommand, log id: 63a9ee2b >2014-04-29 20:20:00,758 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [31ba4f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:20:00,768 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [4f3e9e52] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:20:00,769 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [18044745] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:20:00,770 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [18044745] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 241afb92 >2014-04-29 20:20:00,784 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [18044745] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:20:00,784 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [18044745] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:20:02,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 75670eb1 >2014-04-29 20:20:02,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] Failed in GetHardwareInfoVDS method >2014-04-29 20:20:02,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:20:02,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@40f5c2ae >2014-04-29 20:20:02,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] HostName = 10.34.27.209 >2014-04-29 20:20:02,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:20:02,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] FINISH, GetHardwareInfoVDSCommand, log id: 75670eb1 >2014-04-29 20:20:02,691 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:20:02,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [4dc1db0c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:20:02,922 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-79) [5afcb4a8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:20:02,924 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-79) [6cfd69a0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:20:02,930 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [6cfd69a0] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2e3e85ac >2014-04-29 20:23:02,931 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [6cfd69a0] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:23:02,931 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-79) [6cfd69a0] FINISH, ConnectStorageServerVDSCommand, log id: 2e3e85ac >2014-04-29 20:23:02,931 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-79) [6cfd69a0] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:23:02,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1ebad342 >2014-04-29 20:23:03,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 1ebad342 >2014-04-29 20:23:03,396 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:23:03,407 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-79) [2c4fdfa8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:23:03,408 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [2c4fdfa8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7e60f7cd >2014-04-29 20:23:03,410 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [2c4fdfa8] FINISH, SetVdsStatusVDSCommand, log id: 7e60f7cd >2014-04-29 20:23:03,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [2c4fdfa8] Correlation ID: 2c4fdfa8, Job ID: 6cc99d0c-16bd-494e-8e08-80d74c68a571, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:23:03,424 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [2c4fdfa8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:23:03,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [2c4fdfa8] Correlation ID: 5afcb4a8, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:23:03,430 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [30ddcb8c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:23:03,431 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [2253937a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:23:03,431 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-79) [2253937a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) [458a8012] Autorecovering 1 hosts >2014-04-29 20:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) [458a8012] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5c2b64c9 >2014-04-29 20:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] FINISH, SetVdsStatusVDSCommand, log id: 5c2b64c9 >2014-04-29 20:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3c8ae6da >2014-04-29 20:25:00,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 137e5ddc >2014-04-29 20:25:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] Failed in GetHardwareInfoVDS method >2014-04-29 20:25:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:25:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4d442083 >2014-04-29 20:25:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] HostName = 10.34.27.209 >2014-04-29 20:25:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:25:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [363f2509] FINISH, GetHardwareInfoVDSCommand, log id: 137e5ddc >2014-04-29 20:25:00,708 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [363f2509] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:25:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-85) [3927ea7a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:25:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-85) [46031e23] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:25:00,718 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) [46031e23] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3c8ae6da >2014-04-29 20:25:00,732 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [46031e23] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:25:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [46031e23] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:25:03,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f3793f3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 384c5428 >2014-04-29 20:25:04,021 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f3793f3] Failed in GetHardwareInfoVDS method >2014-04-29 20:25:04,021 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f3793f3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:25:04,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f3793f3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@574f0cff >2014-04-29 20:25:04,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f3793f3] HostName = 10.34.27.209 >2014-04-29 20:25:04,021 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f3793f3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:25:04,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [7f3793f3] FINISH, GetHardwareInfoVDSCommand, log id: 384c5428 >2014-04-29 20:25:04,026 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [7f3793f3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:25:04,034 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [7f3793f3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:25:04,257 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-6) [5d50b25d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:25:04,258 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-6) [2d3aea1e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:25:04,264 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [2d3aea1e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1ddf9daf >2014-04-29 20:28:04,266 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [2d3aea1e] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:28:04,266 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-6) [2d3aea1e] FINISH, ConnectStorageServerVDSCommand, log id: 1ddf9daf >2014-04-29 20:28:04,266 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-6) [2d3aea1e] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:28:04,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 15201ff3 >2014-04-29 20:28:04,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 15201ff3 >2014-04-29 20:28:04,719 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:28:04,730 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-6) [5bb5a0b0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:28:04,731 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [5bb5a0b0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3468ba67 >2014-04-29 20:28:04,733 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [5bb5a0b0] FINISH, SetVdsStatusVDSCommand, log id: 3468ba67 >2014-04-29 20:28:04,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [5bb5a0b0] Correlation ID: 5bb5a0b0, Job ID: 3fae4fc3-fa36-4438-bdc0-b57d10aa89fc, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:28:04,746 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [5bb5a0b0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:28:04,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [5bb5a0b0] Correlation ID: 5d50b25d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:28:04,760 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [77515e49] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:28:04,764 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [4e1b8719] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:28:04,764 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-6) [4e1b8719] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-61) Autorecovering 1 hosts >2014-04-29 20:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-61) Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2b121b7d >2014-04-29 20:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] FINISH, SetVdsStatusVDSCommand, log id: 2b121b7d >2014-04-29 20:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1c363ecd >2014-04-29 20:30:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 339cca47 >2014-04-29 20:30:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] Failed in GetHardwareInfoVDS method >2014-04-29 20:30:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:30:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2a927e06 >2014-04-29 20:30:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] HostName = 10.34.27.209 >2014-04-29 20:30:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:30:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [47d8235d] FINISH, GetHardwareInfoVDSCommand, log id: 339cca47 >2014-04-29 20:30:00,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [47d8235d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:30:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-61) [217612fc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:30:00,725 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-61) [19c7c217] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:30:00,726 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) [19c7c217] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1c363ecd >2014-04-29 20:30:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [19c7c217] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:30:00,743 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [19c7c217] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:30:02,133 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 60b828eb >2014-04-29 20:30:02,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] Failed in GetHardwareInfoVDS method >2014-04-29 20:30:02,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:30:02,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5a868de8 >2014-04-29 20:30:02,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] HostName = 10.34.27.209 >2014-04-29 20:30:02,358 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:30:02,358 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] FINISH, GetHardwareInfoVDSCommand, log id: 60b828eb >2014-04-29 20:30:02,365 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:30:02,374 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [3d5fb7d0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:30:02,597 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-62) [19aabc6d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:30:02,598 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-62) [19e9901d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:30:02,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [19e9901d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 574d24d3 >2014-04-29 20:33:02,605 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [19e9901d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:33:02,605 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [19e9901d] FINISH, ConnectStorageServerVDSCommand, log id: 574d24d3 >2014-04-29 20:33:02,606 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-62) [19e9901d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:33:02,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 17a4d584 >2014-04-29 20:33:03,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 17a4d584 >2014-04-29 20:33:03,069 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:33:03,081 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-62) [43959bd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:33:03,083 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [43959bd] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3ce40a97 >2014-04-29 20:33:03,085 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [43959bd] FINISH, SetVdsStatusVDSCommand, log id: 3ce40a97 >2014-04-29 20:33:03,095 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [43959bd] Correlation ID: 43959bd, Job ID: fa0168b5-80ea-4bf0-99a3-6323928455bd, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:33:03,101 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [43959bd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:33:03,103 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [43959bd] Correlation ID: 19aabc6d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:33:03,108 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-62) [72c31d61] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:33:03,109 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:33:03,110 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-5) [72c4daef] Autorecovering 1 hosts >2014-04-29 20:35:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-5) [72c4daef] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:35:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 19ffe4af >2014-04-29 20:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] FINISH, SetVdsStatusVDSCommand, log id: 19ffe4af >2014-04-29 20:35:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 42f3bbfa >2014-04-29 20:35:00,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 137d2c1b >2014-04-29 20:35:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] Failed in GetHardwareInfoVDS method >2014-04-29 20:35:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:35:00,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5020aee2 >2014-04-29 20:35:00,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] HostName = 10.34.27.209 >2014-04-29 20:35:00,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:35:00,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [198be64d] FINISH, GetHardwareInfoVDSCommand, log id: 137d2c1b >2014-04-29 20:35:00,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [198be64d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:35:00,718 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-5) [d55e028] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:35:00,719 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:35:00,720 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 42f3bbfa >2014-04-29 20:35:00,736 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:35:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:35:03,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 651a810f >2014-04-29 20:35:03,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] Failed in GetHardwareInfoVDS method >2014-04-29 20:35:03,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:35:03,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2f0f33a4 >2014-04-29 20:35:03,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] HostName = 10.34.27.209 >2014-04-29 20:35:03,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:35:03,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] FINISH, GetHardwareInfoVDSCommand, log id: 651a810f >2014-04-29 20:35:03,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:35:03,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [13fd4c2b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:35:03,945 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-38) [7a06f51b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:35:03,946 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-38) [431868b8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:35:03,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [431868b8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c910a3a >2014-04-29 20:36:05,005 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-65) [26e34cd9] Start refreshing all users data >2014-04-29 20:36:07,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) [796995ca] Attempting to update VMs/Templates Ovf. >2014-04-29 20:36:07,275 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) [796995ca] Attempting to update VM OVFs in Data Center NFS >2014-04-29 20:36:07,276 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) [796995ca] Successfully updated VM OVFs in Data Center NFS >2014-04-29 20:36:07,276 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) [796995ca] Attempting to update template OVFs in Data Center NFS >2014-04-29 20:36:07,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) [796995ca] Successfully updated templates OVFs in Data Center NFS >2014-04-29 20:36:07,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) [796995ca] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 20:36:07,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-58) [796995ca] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 20:38:03,954 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [431868b8] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:38:03,954 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [431868b8] FINISH, ConnectStorageServerVDSCommand, log id: 6c910a3a >2014-04-29 20:38:03,954 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-38) [431868b8] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:38:03,962 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 49314ffd >2014-04-29 20:38:04,419 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 49314ffd >2014-04-29 20:38:04,419 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:38:04,430 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-38) [8ce0604] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:38:04,431 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [8ce0604] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3c776e36 >2014-04-29 20:38:04,433 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [8ce0604] FINISH, SetVdsStatusVDSCommand, log id: 3c776e36 >2014-04-29 20:38:04,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [8ce0604] Correlation ID: 8ce0604, Job ID: ed0ad003-fdd5-4fe9-899b-e932457dd818, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:38:04,445 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [8ce0604] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:38:04,447 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [8ce0604] Correlation ID: 7a06f51b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:38:04,451 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-38) [42655d99] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:38:04,452 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-38) [1aa55f29] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:38:04,453 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [1aa55f29] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-34) [67993f0e] Autorecovering 1 hosts >2014-04-29 20:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-34) [67993f0e] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:40:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:40:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:40:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:40:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1ffa8255 >2014-04-29 20:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] FINISH, SetVdsStatusVDSCommand, log id: 1ffa8255 >2014-04-29 20:40:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 62a63543 >2014-04-29 20:40:00,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4ee67257 >2014-04-29 20:40:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] Failed in GetHardwareInfoVDS method >2014-04-29 20:40:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:40:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7b5cd638 >2014-04-29 20:40:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] HostName = 10.34.27.209 >2014-04-29 20:40:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:40:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [5c536287] FINISH, GetHardwareInfoVDSCommand, log id: 4ee67257 >2014-04-29 20:40:00,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [5c536287] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:40:00,730 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [61288da4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:40:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [50ceb9aa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:40:00,732 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) [50ceb9aa] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 62a63543 >2014-04-29 20:40:00,753 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [50ceb9aa] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:40:00,754 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [50ceb9aa] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:40:01,810 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [54b31938] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7b649b2b >2014-04-29 20:40:02,033 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [54b31938] Failed in GetHardwareInfoVDS method >2014-04-29 20:40:02,033 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [54b31938] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:40:02,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [54b31938] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@a076357 >2014-04-29 20:40:02,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [54b31938] HostName = 10.34.27.209 >2014-04-29 20:40:02,033 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [54b31938] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:40:02,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [54b31938] FINISH, GetHardwareInfoVDSCommand, log id: 7b649b2b >2014-04-29 20:40:02,040 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [54b31938] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:40:02,051 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [54b31938] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:40:02,274 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-52) [3fd62d72] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:40:02,276 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-52) [3c2663ec] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:40:02,283 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [3c2663ec] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 20848217 >2014-04-29 20:43:02,284 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [3c2663ec] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:43:02,284 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [3c2663ec] FINISH, ConnectStorageServerVDSCommand, log id: 20848217 >2014-04-29 20:43:02,284 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-52) [3c2663ec] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:43:02,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 25d67d85 >2014-04-29 20:43:02,731 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 25d67d85 >2014-04-29 20:43:02,731 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:43:02,742 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-52) [35768998] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:43:02,743 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [35768998] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3b71ffa >2014-04-29 20:43:02,745 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [35768998] FINISH, SetVdsStatusVDSCommand, log id: 3b71ffa >2014-04-29 20:43:02,751 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [35768998] Correlation ID: 35768998, Job ID: 112e45ac-ef1e-478c-b3e8-f3ade30baa97, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:43:02,756 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [35768998] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:43:02,758 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [35768998] Correlation ID: 3fd62d72, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:43:02,762 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-52) [67b8d007] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:43:02,763 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-52) [897e89b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:43:02,763 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [897e89b] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-32) [61d1fd27] Autorecovering 1 hosts >2014-04-29 20:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-32) [61d1fd27] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 164a2f76 >2014-04-29 20:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] FINISH, SetVdsStatusVDSCommand, log id: 164a2f76 >2014-04-29 20:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 337ba66a >2014-04-29 20:45:00,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 27142b45 >2014-04-29 20:45:00,740 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] Failed in GetHardwareInfoVDS method >2014-04-29 20:45:00,740 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:45:00,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1d24d30e >2014-04-29 20:45:00,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] HostName = 10.34.27.209 >2014-04-29 20:45:00,740 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:45:00,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [646cde9c] FINISH, GetHardwareInfoVDSCommand, log id: 27142b45 >2014-04-29 20:45:00,745 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [646cde9c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:45:00,754 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-32) [76be8c0b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:45:00,755 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:45:00,755 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 337ba66a >2014-04-29 20:45:00,770 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:45:00,771 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:45:03,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 65db1c85 >2014-04-29 20:45:03,341 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] Failed in GetHardwareInfoVDS method >2014-04-29 20:45:03,341 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:45:03,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@142d9401 >2014-04-29 20:45:03,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] HostName = 10.34.27.209 >2014-04-29 20:45:03,341 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:45:03,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [562a1877] FINISH, GetHardwareInfoVDSCommand, log id: 65db1c85 >2014-04-29 20:45:03,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [562a1877] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:45:03,355 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [562a1877] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:45:03,576 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-64) [1b54d416] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:45:03,578 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [46ee5051] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:45:03,584 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [46ee5051] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c1fdec1 >2014-04-29 20:48:03,586 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [46ee5051] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:48:03,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [46ee5051] FINISH, ConnectStorageServerVDSCommand, log id: 6c1fdec1 >2014-04-29 20:48:03,586 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [46ee5051] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:48:03,593 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 13483c6c >2014-04-29 20:48:04,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 13483c6c >2014-04-29 20:48:04,251 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:48:04,261 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-64) [246be46] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:48:04,262 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [246be46] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 8430da2 >2014-04-29 20:48:04,265 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [246be46] FINISH, SetVdsStatusVDSCommand, log id: 8430da2 >2014-04-29 20:48:04,271 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [246be46] Correlation ID: 246be46, Job ID: c66c3157-e62e-4491-8e98-6af09fec5849, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:48:04,276 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [246be46] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:48:04,278 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [246be46] Correlation ID: 1b54d416, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:48:04,282 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [58e56766] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:48:04,283 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [365f6e6d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:48:04,284 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [365f6e6d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-84) [2e21be14] Autorecovering 1 hosts >2014-04-29 20:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-84) [2e21be14] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:50:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7457269 >2014-04-29 20:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] FINISH, SetVdsStatusVDSCommand, log id: 7457269 >2014-04-29 20:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 66bcef2b >2014-04-29 20:50:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1e48000a >2014-04-29 20:50:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] Failed in GetHardwareInfoVDS method >2014-04-29 20:50:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:50:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@8faa00e >2014-04-29 20:50:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] HostName = 10.34.27.209 >2014-04-29 20:50:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:50:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [eb82e44] FINISH, GetHardwareInfoVDSCommand, log id: 1e48000a >2014-04-29 20:50:00,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [eb82e44] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:50:00,713 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [4d7773d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:50:00,714 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [2f537f0b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:50:00,715 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [2f537f0b] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 66bcef2b >2014-04-29 20:50:00,729 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [2f537f0b] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:50:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [2f537f0b] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:50:01,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4b08a58d >2014-04-29 20:50:01,880 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] Failed in GetHardwareInfoVDS method >2014-04-29 20:50:01,880 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:50:01,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@24acf277 >2014-04-29 20:50:01,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] HostName = 10.34.27.209 >2014-04-29 20:50:01,880 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:50:01,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] FINISH, GetHardwareInfoVDSCommand, log id: 4b08a58d >2014-04-29 20:50:01,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:50:01,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [44d1ed6a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:50:02,117 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-10) [4a2152bb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:50:02,118 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-10) [b254fb0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:50:02,125 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-10) [b254fb0] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 56a2b033 >2014-04-29 20:53:02,127 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-10) [b254fb0] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:53:02,127 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-10) [b254fb0] FINISH, ConnectStorageServerVDSCommand, log id: 56a2b033 >2014-04-29 20:53:02,127 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-10) [b254fb0] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:53:02,130 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 2685bdaf >2014-04-29 20:53:02,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 2685bdaf >2014-04-29 20:53:02,570 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:53:02,583 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-10) [6b3b6970] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:53:02,584 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [6b3b6970] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 62c2d34e >2014-04-29 20:53:02,587 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [6b3b6970] FINISH, SetVdsStatusVDSCommand, log id: 62c2d34e >2014-04-29 20:53:02,594 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [6b3b6970] Correlation ID: 6b3b6970, Job ID: 53ded1e7-b7ca-42ea-9931-101ce23f9d12, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:53:02,604 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [6b3b6970] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:53:02,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [6b3b6970] Correlation ID: 4a2152bb, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:53:02,610 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-10) [4b90a9a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:53:02,611 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-10) [5420242e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:53:02,612 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) [5420242e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 20:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [5b3461c7] Autorecovering 1 hosts >2014-04-29 20:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-97) [5b3461c7] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 20:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 20:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 20:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2cde6374 >2014-04-29 20:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] FINISH, SetVdsStatusVDSCommand, log id: 2cde6374 >2014-04-29 20:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 44b1f8cb >2014-04-29 20:55:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4c2e8241 >2014-04-29 20:55:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] Failed in GetHardwareInfoVDS method >2014-04-29 20:55:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:55:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@169a07a >2014-04-29 20:55:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] HostName = 10.34.27.209 >2014-04-29 20:55:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:55:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] FINISH, GetHardwareInfoVDSCommand, log id: 4c2e8241 >2014-04-29 20:55:00,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [6a5f7d43] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:55:00,727 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-97) [23f2d8ea] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:55:00,729 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:55:00,730 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 44b1f8cb >2014-04-29 20:55:00,749 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 20:55:00,750 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 20:55:02,982 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 59a3b94d >2014-04-29 20:55:03,206 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] Failed in GetHardwareInfoVDS method >2014-04-29 20:55:03,206 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:55:03,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@bda9c35 >2014-04-29 20:55:03,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] HostName = 10.34.27.209 >2014-04-29 20:55:03,206 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 20:55:03,206 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] FINISH, GetHardwareInfoVDSCommand, log id: 59a3b94d >2014-04-29 20:55:03,211 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 20:55:03,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [7c6e4cca] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 20:55:03,441 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-19) [3c377b31] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:55:03,442 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [25ef567c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 20:55:03,449 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [25ef567c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 553c11a0 >2014-04-29 20:58:03,451 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [25ef567c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 20:58:03,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [25ef567c] FINISH, ConnectStorageServerVDSCommand, log id: 553c11a0 >2014-04-29 20:58:03,451 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [25ef567c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 20:58:03,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5fbe156b >2014-04-29 20:58:03,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 5fbe156b >2014-04-29 20:58:03,899 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 20:58:03,946 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-19) [138b83dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:58:03,947 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [138b83dc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7a1a0a9c >2014-04-29 20:58:03,950 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [138b83dc] FINISH, SetVdsStatusVDSCommand, log id: 7a1a0a9c >2014-04-29 20:58:03,956 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [138b83dc] Correlation ID: 138b83dc, Job ID: ec16f3e6-e79c-4109-a1be-132e7a86ae1c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 20:58:03,963 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [138b83dc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 20:58:03,965 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [138b83dc] Correlation ID: 3c377b31, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 20:58:03,971 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-19) [604aa3fd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:58:03,972 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-19) [3bf9c883] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 20:58:03,972 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [3bf9c883] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) [1d0ba8da] Autorecovering 1 hosts >2014-04-29 21:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) [1d0ba8da] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:00:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:00:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:00:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:00:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 348af588 >2014-04-29 21:00:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] FINISH, SetVdsStatusVDSCommand, log id: 348af588 >2014-04-29 21:00:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2dde7dfb >2014-04-29 21:00:00,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 47111a2e >2014-04-29 21:00:00,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] Failed in GetHardwareInfoVDS method >2014-04-29 21:00:00,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:00:00,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@33cde675 >2014-04-29 21:00:00,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] HostName = 10.34.27.209 >2014-04-29 21:00:00,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:00:00,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [20bae9be] FINISH, GetHardwareInfoVDSCommand, log id: 47111a2e >2014-04-29 21:00:00,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [20bae9be] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:00:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-90) [7d526268] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:00:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-90) [49239c0f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:00:00,718 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) [49239c0f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2dde7dfb >2014-04-29 21:00:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [49239c0f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:00:00,732 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [49239c0f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:00:01,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [246429fc] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 42591235 >2014-04-29 21:00:01,546 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [246429fc] Failed in GetHardwareInfoVDS method >2014-04-29 21:00:01,546 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [246429fc] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:00:01,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [246429fc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@49dc9f7c >2014-04-29 21:00:01,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [246429fc] HostName = 10.34.27.209 >2014-04-29 21:00:01,546 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [246429fc] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:00:01,546 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [246429fc] FINISH, GetHardwareInfoVDSCommand, log id: 42591235 >2014-04-29 21:00:01,551 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [246429fc] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:00:01,560 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [246429fc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:00:01,782 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-15) [7467becc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:00:01,783 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-15) [4ad40f06] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:00:01,790 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [4ad40f06] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1cda4fd >2014-04-29 21:03:01,792 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [4ad40f06] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:03:01,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [4ad40f06] FINISH, ConnectStorageServerVDSCommand, log id: 1cda4fd >2014-04-29 21:03:01,792 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-15) [4ad40f06] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:03:01,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4fb2da72 >2014-04-29 21:03:02,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 4fb2da72 >2014-04-29 21:03:02,246 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:03:02,258 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-15) [374e5556] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:03:02,260 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [374e5556] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 771d7af7 >2014-04-29 21:03:02,263 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [374e5556] FINISH, SetVdsStatusVDSCommand, log id: 771d7af7 >2014-04-29 21:03:02,277 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [374e5556] Correlation ID: 374e5556, Job ID: a744b930-59c5-4703-943c-47f422cb0da7, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:03:02,286 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [374e5556] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:03:02,288 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [374e5556] Correlation ID: 7467becc, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:03:02,293 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-15) [76571190] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:03:02,294 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-15) [403277f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:03:02,295 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [403277f3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-34) [50ceb9aa] Autorecovering 1 hosts >2014-04-29 21:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-34) [50ceb9aa] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 215ff76d >2014-04-29 21:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] FINISH, SetVdsStatusVDSCommand, log id: 215ff76d >2014-04-29 21:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 55c11be4 >2014-04-29 21:05:00,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2f00bdea >2014-04-29 21:05:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] Failed in GetHardwareInfoVDS method >2014-04-29 21:05:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:05:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3183e77e >2014-04-29 21:05:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] HostName = 10.34.27.209 >2014-04-29 21:05:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:05:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [66aeb986] FINISH, GetHardwareInfoVDSCommand, log id: 2f00bdea >2014-04-29 21:05:00,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [66aeb986] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:05:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [a270f30] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:05:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [20b6ccf4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:05:00,724 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) [20b6ccf4] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 55c11be4 >2014-04-29 21:05:00,739 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [20b6ccf4] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:05:00,740 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [20b6ccf4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:05:02,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [665b237b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 59892a49 >2014-04-29 21:05:02,883 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [665b237b] Failed in GetHardwareInfoVDS method >2014-04-29 21:05:02,883 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [665b237b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:05:02,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [665b237b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3c374c8b >2014-04-29 21:05:02,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [665b237b] HostName = 10.34.27.209 >2014-04-29 21:05:02,883 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [665b237b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:05:02,883 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [665b237b] FINISH, GetHardwareInfoVDSCommand, log id: 59892a49 >2014-04-29 21:05:02,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [665b237b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:05:02,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [665b237b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:05:03,122 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-59) [1a5f76af] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:05:03,123 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-59) [2109d461] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:05:03,129 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [2109d461] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5b4bde53 >2014-04-29 21:08:03,131 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [2109d461] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:08:03,131 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [2109d461] FINISH, ConnectStorageServerVDSCommand, log id: 5b4bde53 >2014-04-29 21:08:03,131 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-59) [2109d461] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:08:03,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 47c93572 >2014-04-29 21:08:03,603 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 47c93572 >2014-04-29 21:08:03,603 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:08:03,617 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-59) [740ad243] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:08:03,619 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [740ad243] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2db9b593 >2014-04-29 21:08:03,621 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [740ad243] FINISH, SetVdsStatusVDSCommand, log id: 2db9b593 >2014-04-29 21:08:03,628 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [740ad243] Correlation ID: 740ad243, Job ID: 3d2e6698-1339-49c8-91ee-d6c30862dc79, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:08:03,636 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [740ad243] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:08:03,639 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [740ad243] Correlation ID: 1a5f76af, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:08:03,643 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-59) [1668864c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:08:03,644 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-59) [4974a741] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:08:03,645 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) [4974a741] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) [5586a962] Autorecovering 1 hosts >2014-04-29 21:10:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) [5586a962] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:10:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:10:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:10:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:10:00,003 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:10:00,006 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2469ab40 >2014-04-29 21:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] FINISH, SetVdsStatusVDSCommand, log id: 2469ab40 >2014-04-29 21:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 317e6812 >2014-04-29 21:10:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 675f9b41 >2014-04-29 21:10:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] Failed in GetHardwareInfoVDS method >2014-04-29 21:10:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:10:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@35c9fd6c >2014-04-29 21:10:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] HostName = 10.34.27.209 >2014-04-29 21:10:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:10:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] FINISH, GetHardwareInfoVDSCommand, log id: 675f9b41 >2014-04-29 21:10:00,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [2b1b8ab4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:10:00,711 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [61b7b101] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:10:00,712 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:10:00,713 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 317e6812 >2014-04-29 21:10:00,729 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:10:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:10:04,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5cddc538 >2014-04-29 21:10:04,224 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] Failed in GetHardwareInfoVDS method >2014-04-29 21:10:04,224 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:10:04,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@cf9e4cb >2014-04-29 21:10:04,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] HostName = 10.34.27.209 >2014-04-29 21:10:04,224 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:10:04,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [6dab1724] FINISH, GetHardwareInfoVDSCommand, log id: 5cddc538 >2014-04-29 21:10:04,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [6dab1724] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:10:04,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [6dab1724] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:10:04,460 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-24) [51a01560] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:10:04,461 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [59a45558] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:10:04,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [59a45558] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6fecb060 >2014-04-29 21:13:04,469 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [59a45558] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:13:04,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-24) [59a45558] FINISH, ConnectStorageServerVDSCommand, log id: 6fecb060 >2014-04-29 21:13:04,469 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-24) [59a45558] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:13:04,472 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 492ebcb2 >2014-04-29 21:13:04,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 492ebcb2 >2014-04-29 21:13:04,932 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:13:04,945 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-24) [59c65807] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:13:04,946 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [59c65807] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3c637801 >2014-04-29 21:13:04,948 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [59c65807] FINISH, SetVdsStatusVDSCommand, log id: 3c637801 >2014-04-29 21:13:04,954 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [59c65807] Correlation ID: 59c65807, Job ID: 996d7471-9073-411c-a108-ae0041c31405, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:13:04,961 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [59c65807] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:13:04,963 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [59c65807] Correlation ID: 51a01560, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:13:04,967 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-24) [3d9f786a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:13:04,972 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-24) [5904254f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:13:04,973 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-24) [5904254f] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-84) [2f537f0b] Autorecovering 1 hosts >2014-04-29 21:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-84) [2f537f0b] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 342509a4 >2014-04-29 21:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] FINISH, SetVdsStatusVDSCommand, log id: 342509a4 >2014-04-29 21:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3fce7b98 >2014-04-29 21:15:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 46f869ce >2014-04-29 21:15:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] Failed in GetHardwareInfoVDS method >2014-04-29 21:15:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:15:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2fa51d72 >2014-04-29 21:15:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] HostName = 10.34.27.209 >2014-04-29 21:15:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:15:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] FINISH, GetHardwareInfoVDSCommand, log id: 46f869ce >2014-04-29 21:15:00,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [6a2e2c95] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:15:00,713 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [23b1d4f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:15:00,714 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:15:00,714 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3fce7b98 >2014-04-29 21:15:00,728 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:15:00,729 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:15:02,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [22eeda0a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 412f75f1 >2014-04-29 21:15:02,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [22eeda0a] Failed in GetHardwareInfoVDS method >2014-04-29 21:15:02,553 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [22eeda0a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:15:02,553 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [22eeda0a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@12e842a6 >2014-04-29 21:15:02,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [22eeda0a] HostName = 10.34.27.209 >2014-04-29 21:15:02,554 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [22eeda0a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:15:02,554 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [22eeda0a] FINISH, GetHardwareInfoVDSCommand, log id: 412f75f1 >2014-04-29 21:15:02,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [22eeda0a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:15:02,569 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [22eeda0a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:15:02,791 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-47) [6ac0cfef] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:15:02,793 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-47) [34b6b011] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:15:02,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-47) [34b6b011] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 18dc9ca4 >2014-04-29 21:18:02,800 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-47) [34b6b011] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:18:02,800 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-47) [34b6b011] FINISH, ConnectStorageServerVDSCommand, log id: 18dc9ca4 >2014-04-29 21:18:02,801 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-47) [34b6b011] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:18:02,804 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4a8a20e1 >2014-04-29 21:18:03,246 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 4a8a20e1 >2014-04-29 21:18:03,246 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:18:03,262 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-47) [f78f638] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:18:03,263 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [f78f638] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 796450b4 >2014-04-29 21:18:03,265 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [f78f638] FINISH, SetVdsStatusVDSCommand, log id: 796450b4 >2014-04-29 21:18:03,271 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [f78f638] Correlation ID: f78f638, Job ID: 312886b0-bfc0-4399-936e-3c25f5b3604b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:18:03,278 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [f78f638] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:18:03,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [f78f638] Correlation ID: 6ac0cfef, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:18:03,285 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-47) [7b1dfb5c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:18:03,286 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-47) [63374716] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:18:03,287 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-47) [63374716] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) [116711a] Autorecovering 1 hosts >2014-04-29 21:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) [116711a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6f0155a6 >2014-04-29 21:20:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] FINISH, SetVdsStatusVDSCommand, log id: 6f0155a6 >2014-04-29 21:20:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 58d6d5ff >2014-04-29 21:20:00,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4488a30b >2014-04-29 21:20:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] Failed in GetHardwareInfoVDS method >2014-04-29 21:20:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:20:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@66884a53 >2014-04-29 21:20:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] HostName = 10.34.27.209 >2014-04-29 21:20:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:20:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [710eb468] FINISH, GetHardwareInfoVDSCommand, log id: 4488a30b >2014-04-29 21:20:00,721 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [710eb468] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:20:00,730 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [6d32e2a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:20:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-87) [6f98499a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:20:00,731 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [6f98499a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 58d6d5ff >2014-04-29 21:20:00,745 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [6f98499a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:20:00,749 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [6f98499a] Correlation ID: 710eb468, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was autorecovered. >2014-04-29 21:20:00,749 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [6f98499a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:20:03,638 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 33014e4 >2014-04-29 21:20:03,861 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] Failed in GetHardwareInfoVDS method >2014-04-29 21:20:03,861 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:20:03,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@49ab865e >2014-04-29 21:20:03,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] HostName = 10.34.27.209 >2014-04-29 21:20:03,861 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:20:03,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] FINISH, GetHardwareInfoVDSCommand, log id: 33014e4 >2014-04-29 21:20:03,866 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:20:03,877 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [6d9b1a09] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:20:04,100 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-62) [3e43098f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:20:04,101 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-62) [44963b34] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:20:04,108 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [44963b34] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 53882c78 >2014-04-29 21:23:04,110 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [44963b34] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:23:04,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [44963b34] FINISH, ConnectStorageServerVDSCommand, log id: 53882c78 >2014-04-29 21:23:04,110 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-62) [44963b34] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:23:04,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 72ab2ab6 >2014-04-29 21:23:04,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 72ab2ab6 >2014-04-29 21:23:04,560 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:23:04,573 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-62) [becc19d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:23:04,574 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [becc19d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7a3a6394 >2014-04-29 21:23:04,578 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [becc19d] FINISH, SetVdsStatusVDSCommand, log id: 7a3a6394 >2014-04-29 21:23:04,587 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [becc19d] Correlation ID: becc19d, Job ID: a1ad8d3f-df1b-4278-978a-98bc67efbed0, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:23:04,594 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [becc19d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:23:04,597 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [becc19d] Correlation ID: 3e43098f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:23:04,602 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-62) [7eea400f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:23:04,603 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-62) [383fdecb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:23:04,604 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) [383fdecb] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-45) [204cd92a] Autorecovering 1 hosts >2014-04-29 21:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-45) [204cd92a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4974de41 >2014-04-29 21:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] FINISH, SetVdsStatusVDSCommand, log id: 4974de41 >2014-04-29 21:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 48eca462 >2014-04-29 21:25:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 284a1795 >2014-04-29 21:25:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] Failed in GetHardwareInfoVDS method >2014-04-29 21:25:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:25:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@18524bdd >2014-04-29 21:25:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] HostName = 10.34.27.209 >2014-04-29 21:25:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:25:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [47d343e] FINISH, GetHardwareInfoVDSCommand, log id: 284a1795 >2014-04-29 21:25:00,708 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [47d343e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:25:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [1725a6b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:25:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:25:00,718 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 48eca462 >2014-04-29 21:25:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:25:00,732 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:25:01,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 570c098f >2014-04-29 21:25:02,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] Failed in GetHardwareInfoVDS method >2014-04-29 21:25:02,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:25:02,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5747b7ee >2014-04-29 21:25:02,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] HostName = 10.34.27.209 >2014-04-29 21:25:02,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:25:02,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [1fa06910] FINISH, GetHardwareInfoVDSCommand, log id: 570c098f >2014-04-29 21:25:02,194 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [1fa06910] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:25:02,205 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [1fa06910] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:25:02,427 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-46) [3a2ad9f1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:25:02,428 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-46) [437d1a25] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:25:02,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [437d1a25] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 72d239fd >2014-04-29 21:28:02,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [437d1a25] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:28:02,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [437d1a25] FINISH, ConnectStorageServerVDSCommand, log id: 72d239fd >2014-04-29 21:28:02,436 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-46) [437d1a25] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:28:02,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 43567b3e >2014-04-29 21:28:02,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 43567b3e >2014-04-29 21:28:02,880 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:28:02,890 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-46) [64126223] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:28:02,891 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [64126223] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2676d900 >2014-04-29 21:28:02,893 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [64126223] FINISH, SetVdsStatusVDSCommand, log id: 2676d900 >2014-04-29 21:28:02,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [64126223] Correlation ID: 64126223, Job ID: a2657580-6655-426d-827e-d9d51d12aa0a, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:28:02,906 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [64126223] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:28:02,908 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [64126223] Correlation ID: 3a2ad9f1, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:28:02,912 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-46) [45ffd187] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:28:02,913 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-46) [5458a17c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:28:02,913 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) [5458a17c] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [5f62d99b] Autorecovering 1 hosts >2014-04-29 21:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [5f62d99b] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 671300ed >2014-04-29 21:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] FINISH, SetVdsStatusVDSCommand, log id: 671300ed >2014-04-29 21:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6fe6b9f5 >2014-04-29 21:30:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 67891de1 >2014-04-29 21:30:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] Failed in GetHardwareInfoVDS method >2014-04-29 21:30:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:30:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5b08d3e1 >2014-04-29 21:30:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] HostName = 10.34.27.209 >2014-04-29 21:30:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:30:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] FINISH, GetHardwareInfoVDSCommand, log id: 67891de1 >2014-04-29 21:30:00,719 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [5c0a6ef3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:30:00,772 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-33) [5b02f7e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:30:00,773 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-33) [582c9313] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:30:00,774 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) [582c9313] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6fe6b9f5 >2014-04-29 21:30:00,799 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [582c9313] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:30:00,801 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [582c9313] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:30:03,292 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4d3468b6 >2014-04-29 21:30:03,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] Failed in GetHardwareInfoVDS method >2014-04-29 21:30:03,516 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:30:03,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2bc6ac53 >2014-04-29 21:30:03,516 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] HostName = 10.34.27.209 >2014-04-29 21:30:03,517 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:30:03,517 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] FINISH, GetHardwareInfoVDSCommand, log id: 4d3468b6 >2014-04-29 21:30:03,523 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:30:03,531 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [6c6a9b4e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:30:03,760 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-80) [7a62758d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:30:03,767 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [7fca52e6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:30:03,773 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [7fca52e6] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 15f05fc6 >2014-04-29 21:33:03,775 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [7fca52e6] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:33:03,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [7fca52e6] FINISH, ConnectStorageServerVDSCommand, log id: 15f05fc6 >2014-04-29 21:33:03,775 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [7fca52e6] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:33:03,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3b41fbec >2014-04-29 21:33:04,219 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3b41fbec >2014-04-29 21:33:04,219 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:33:04,231 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-80) [1ac1bcf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:33:04,232 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [1ac1bcf] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2c224d1 >2014-04-29 21:33:04,235 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [1ac1bcf] FINISH, SetVdsStatusVDSCommand, log id: 2c224d1 >2014-04-29 21:33:04,241 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [1ac1bcf] Correlation ID: 1ac1bcf, Job ID: 46be718f-204b-476d-861d-bd235d446418, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:33:04,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [1ac1bcf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:33:04,250 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [1ac1bcf] Correlation ID: 7a62758d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:33:04,255 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-80) [6455d14b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:33:04,256 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-80) [21efa807] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:33:04,257 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) [21efa807] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-22) [2d93263f] Autorecovering 1 hosts >2014-04-29 21:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-22) [2d93263f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 58234b63 >2014-04-29 21:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] FINISH, SetVdsStatusVDSCommand, log id: 58234b63 >2014-04-29 21:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3298f90c >2014-04-29 21:35:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 25227d13 >2014-04-29 21:35:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] Failed in GetHardwareInfoVDS method >2014-04-29 21:35:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:35:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@cdc5523 >2014-04-29 21:35:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] HostName = 10.34.27.209 >2014-04-29 21:35:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:35:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [788a558f] FINISH, GetHardwareInfoVDSCommand, log id: 25227d13 >2014-04-29 21:35:00,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [788a558f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:35:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-22) [2ec6f91d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:35:00,718 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:35:00,718 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3298f90c >2014-04-29 21:35:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:35:00,734 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:35:01,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [2fd2615] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2ff81acb >2014-04-29 21:35:01,833 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [2fd2615] Failed in GetHardwareInfoVDS method >2014-04-29 21:35:01,833 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [2fd2615] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:35:01,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [2fd2615] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7ce6f62d >2014-04-29 21:35:01,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [2fd2615] HostName = 10.34.27.209 >2014-04-29 21:35:01,833 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [2fd2615] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:35:01,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [2fd2615] FINISH, GetHardwareInfoVDSCommand, log id: 2ff81acb >2014-04-29 21:35:01,838 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [2fd2615] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:35:01,848 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [2fd2615] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:35:02,070 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-28) [e7c7354] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:35:02,071 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-28) [7a4c5d4e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:35:02,077 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [7a4c5d4e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 74b4fd28 >2014-04-29 21:36:05,007 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) [46031e23] Start refreshing all users data >2014-04-29 21:36:07,277 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Attempting to update VMs/Templates Ovf. >2014-04-29 21:36:07,278 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Attempting to update VM OVFs in Data Center NFS >2014-04-29 21:36:07,279 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Successfully updated VM OVFs in Data Center NFS >2014-04-29 21:36:07,279 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Attempting to update template OVFs in Data Center NFS >2014-04-29 21:36:07,279 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Successfully updated templates OVFs in Data Center NFS >2014-04-29 21:36:07,279 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 21:36:07,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 21:38:02,079 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [7a4c5d4e] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:38:02,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [7a4c5d4e] FINISH, ConnectStorageServerVDSCommand, log id: 74b4fd28 >2014-04-29 21:38:02,079 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-28) [7a4c5d4e] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:38:02,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3afbeb27 >2014-04-29 21:38:02,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 3afbeb27 >2014-04-29 21:38:02,532 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:38:02,543 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-28) [401460de] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:38:02,544 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [401460de] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6f9241e7 >2014-04-29 21:38:02,546 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [401460de] FINISH, SetVdsStatusVDSCommand, log id: 6f9241e7 >2014-04-29 21:38:02,552 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [401460de] Correlation ID: 401460de, Job ID: 6d498415-67c9-486e-86ff-d95df4163ee8, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:38:02,557 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [401460de] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:38:02,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [401460de] Correlation ID: e7c7354, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:38:02,563 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-28) [6dc39419] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:38:02,564 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-28) [3f0c5833] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:38:02,565 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [3f0c5833] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:40:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Autorecovering 1 hosts >2014-04-29 21:40:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-3) [7a0debcc] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:40:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:40:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:40:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:40:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 41a569b6 >2014-04-29 21:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] FINISH, SetVdsStatusVDSCommand, log id: 41a569b6 >2014-04-29 21:40:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 645b8a29 >2014-04-29 21:40:00,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4f65229b >2014-04-29 21:40:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] Failed in GetHardwareInfoVDS method >2014-04-29 21:40:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:40:00,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1669b0ca >2014-04-29 21:40:00,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] HostName = 10.34.27.209 >2014-04-29 21:40:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:40:00,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] FINISH, GetHardwareInfoVDSCommand, log id: 4f65229b >2014-04-29 21:40:00,712 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [3f1fef7e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:40:00,721 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-3) [60d8b076] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:40:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:40:00,722 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 645b8a29 >2014-04-29 21:40:00,736 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:40:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:40:02,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [26573b54] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 32eb0915 >2014-04-29 21:40:03,151 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [26573b54] Failed in GetHardwareInfoVDS method >2014-04-29 21:40:03,151 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [26573b54] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:40:03,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [26573b54] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@64a15bd0 >2014-04-29 21:40:03,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [26573b54] HostName = 10.34.27.209 >2014-04-29 21:40:03,151 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [26573b54] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:40:03,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [26573b54] FINISH, GetHardwareInfoVDSCommand, log id: 32eb0915 >2014-04-29 21:40:03,156 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [26573b54] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:40:03,165 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [26573b54] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:40:03,387 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-98) [55358778] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:40:03,389 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [fa912c8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:40:03,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [fa912c8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 68e906c4 >2014-04-29 21:43:03,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [fa912c8] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:43:03,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [fa912c8] FINISH, ConnectStorageServerVDSCommand, log id: 68e906c4 >2014-04-29 21:43:03,398 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [fa912c8] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:43:03,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1cf39b4d >2014-04-29 21:43:03,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 1cf39b4d >2014-04-29 21:43:03,851 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:43:03,915 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-98) [77f76c3a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:43:03,916 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [77f76c3a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4f34d299 >2014-04-29 21:43:03,927 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [77f76c3a] FINISH, SetVdsStatusVDSCommand, log id: 4f34d299 >2014-04-29 21:43:03,937 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [77f76c3a] Correlation ID: 77f76c3a, Job ID: 6e42ee23-42bb-4e33-811d-f7c527ef540d, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:43:03,945 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [77f76c3a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:43:03,947 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [77f76c3a] Correlation ID: 55358778, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:43:03,951 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-98) [238145d6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:43:03,952 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-98) [4f1248ed] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:43:03,953 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [4f1248ed] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) [6f98499a] Autorecovering 1 hosts >2014-04-29 21:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) [6f98499a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4ec110db >2014-04-29 21:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] FINISH, SetVdsStatusVDSCommand, log id: 4ec110db >2014-04-29 21:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 173de58e >2014-04-29 21:45:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3a5b1a1a >2014-04-29 21:45:00,696 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] Failed in GetHardwareInfoVDS method >2014-04-29 21:45:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:45:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@60bdc2e1 >2014-04-29 21:45:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] HostName = 10.34.27.209 >2014-04-29 21:45:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:45:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] FINISH, GetHardwareInfoVDSCommand, log id: 3a5b1a1a >2014-04-29 21:45:00,703 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [64b8b5ca] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:45:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [3f5f9dc7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:45:00,726 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-87) [21d92d5f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:45:00,726 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [21d92d5f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 173de58e >2014-04-29 21:45:00,748 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [21d92d5f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:45:00,749 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [21d92d5f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:45:01,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [5e6a378] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: e606a4e >2014-04-29 21:45:01,527 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [5e6a378] Failed in GetHardwareInfoVDS method >2014-04-29 21:45:01,527 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [5e6a378] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:45:01,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [5e6a378] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5be32e41 >2014-04-29 21:45:01,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [5e6a378] HostName = 10.34.27.209 >2014-04-29 21:45:01,527 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [5e6a378] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:45:01,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [5e6a378] FINISH, GetHardwareInfoVDSCommand, log id: e606a4e >2014-04-29 21:45:01,532 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [5e6a378] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:45:01,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [5e6a378] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:45:01,763 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-50) [17508bf4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:45:01,765 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [4e743799] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:45:01,772 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [4e743799] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 702bb03f >2014-04-29 21:48:01,774 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [4e743799] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:48:01,774 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [4e743799] FINISH, ConnectStorageServerVDSCommand, log id: 702bb03f >2014-04-29 21:48:01,774 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [4e743799] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:48:01,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: ed31475 >2014-04-29 21:48:02,221 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: ed31475 >2014-04-29 21:48:02,221 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:48:02,234 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-50) [531c745a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:48:02,236 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [531c745a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2c6b88f5 >2014-04-29 21:48:02,238 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [531c745a] FINISH, SetVdsStatusVDSCommand, log id: 2c6b88f5 >2014-04-29 21:48:02,245 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [531c745a] Correlation ID: 531c745a, Job ID: 9ffe8b19-686d-4736-923d-f3bf9ef3332e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:48:02,258 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [531c745a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:48:02,260 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [531c745a] Correlation ID: 17508bf4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:48:02,266 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [7093269f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:48:02,267 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [6600b25d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:48:02,268 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [6600b25d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-72) [7b8d2f5d] Autorecovering 1 hosts >2014-04-29 21:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-72) [7b8d2f5d] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 511b9667 >2014-04-29 21:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] FINISH, SetVdsStatusVDSCommand, log id: 511b9667 >2014-04-29 21:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5a716ce1 >2014-04-29 21:50:00,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7f0a4970 >2014-04-29 21:50:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] Failed in GetHardwareInfoVDS method >2014-04-29 21:50:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:50:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7c5b9938 >2014-04-29 21:50:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] HostName = 10.34.27.209 >2014-04-29 21:50:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:50:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] FINISH, GetHardwareInfoVDSCommand, log id: 7f0a4970 >2014-04-29 21:50:00,729 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [6d74cfa0] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:50:00,777 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-72) [2f066c8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:50:00,778 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:50:00,779 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5a716ce1 >2014-04-29 21:50:00,806 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:50:00,807 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:50:02,627 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 48c63db5 >2014-04-29 21:50:02,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] Failed in GetHardwareInfoVDS method >2014-04-29 21:50:02,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:50:02,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4efd5b66 >2014-04-29 21:50:02,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] HostName = 10.34.27.209 >2014-04-29 21:50:02,850 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:50:02,850 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] FINISH, GetHardwareInfoVDSCommand, log id: 48c63db5 >2014-04-29 21:50:02,856 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:50:02,867 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [32b6a4cb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:50:03,089 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-11) [ee7d68b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:50:03,090 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-11) [2008a307] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:50:03,096 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [2008a307] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 149238d0 >2014-04-29 21:53:03,098 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [2008a307] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:53:03,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-11) [2008a307] FINISH, ConnectStorageServerVDSCommand, log id: 149238d0 >2014-04-29 21:53:03,098 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-11) [2008a307] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:53:03,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5ff6e23e >2014-04-29 21:53:03,549 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 5ff6e23e >2014-04-29 21:53:03,549 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:53:03,560 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-11) [4dc40210] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:53:03,561 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [4dc40210] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: b27c2b6 >2014-04-29 21:53:03,563 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [4dc40210] FINISH, SetVdsStatusVDSCommand, log id: b27c2b6 >2014-04-29 21:53:03,569 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [4dc40210] Correlation ID: 4dc40210, Job ID: fd2399b7-2b3c-44ff-877e-cff525d72fd3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:53:03,576 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [4dc40210] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:53:03,578 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [4dc40210] Correlation ID: ee7d68b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:53:03,582 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-11) [6553babe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:53:03,583 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-11) [4f4ccdd1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:53:03,583 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-11) [4f4ccdd1] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 21:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-67) [65d81bca] Autorecovering 1 hosts >2014-04-29 21:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-67) [65d81bca] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 21:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 21:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 21:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5cc50801 >2014-04-29 21:55:00,031 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] FINISH, SetVdsStatusVDSCommand, log id: 5cc50801 >2014-04-29 21:55:00,058 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 69bd794e >2014-04-29 21:55:00,520 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4bafed25 >2014-04-29 21:55:00,742 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] Failed in GetHardwareInfoVDS method >2014-04-29 21:55:00,742 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:55:00,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@afa3ef5 >2014-04-29 21:55:00,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] HostName = 10.34.27.209 >2014-04-29 21:55:00,742 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:55:00,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [7551de01] FINISH, GetHardwareInfoVDSCommand, log id: 4bafed25 >2014-04-29 21:55:00,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [7551de01] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:55:00,762 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-67) [6b410328] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:55:00,763 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-67) [21498e1e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:55:00,764 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) [21498e1e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 69bd794e >2014-04-29 21:55:00,779 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [21498e1e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 21:55:00,780 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [21498e1e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 21:55:03,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4ba147c4 >2014-04-29 21:55:04,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] Failed in GetHardwareInfoVDS method >2014-04-29 21:55:04,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:55:04,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@496ebba6 >2014-04-29 21:55:04,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] HostName = 10.34.27.209 >2014-04-29 21:55:04,171 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 21:55:04,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [7da9be69] FINISH, GetHardwareInfoVDSCommand, log id: 4ba147c4 >2014-04-29 21:55:04,175 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [7da9be69] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 21:55:04,184 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [7da9be69] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 21:55:04,407 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-21) [7f94954] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:55:04,408 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-21) [68067ef2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 21:55:04,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [68067ef2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 16334ca1 >2014-04-29 21:58:04,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [68067ef2] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 21:58:04,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-21) [68067ef2] FINISH, ConnectStorageServerVDSCommand, log id: 16334ca1 >2014-04-29 21:58:04,416 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-21) [68067ef2] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 21:58:04,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4b6819cd >2014-04-29 21:58:04,860 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 4b6819cd >2014-04-29 21:58:04,860 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 21:58:04,872 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-21) [26aa3362] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:58:04,873 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [26aa3362] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 98acef >2014-04-29 21:58:04,876 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [26aa3362] FINISH, SetVdsStatusVDSCommand, log id: 98acef >2014-04-29 21:58:04,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [26aa3362] Correlation ID: 26aa3362, Job ID: 943691f4-d56e-4f93-8656-0970961dcdbd, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 21:58:04,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [26aa3362] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 21:58:04,892 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [26aa3362] Correlation ID: 7f94954, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 21:58:04,896 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-21) [33f22f5e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:58:04,897 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-21) [413d6449] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 21:58:04,898 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-21) [413d6449] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [64748dcc] Autorecovering 1 hosts >2014-04-29 22:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [64748dcc] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:00:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3083622f >2014-04-29 22:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] FINISH, SetVdsStatusVDSCommand, log id: 3083622f >2014-04-29 22:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 11819362 >2014-04-29 22:00:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1304de57 >2014-04-29 22:00:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] Failed in GetHardwareInfoVDS method >2014-04-29 22:00:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:00:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5832126d >2014-04-29 22:00:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] HostName = 10.34.27.209 >2014-04-29 22:00:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:00:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [747f76ba] FINISH, GetHardwareInfoVDSCommand, log id: 1304de57 >2014-04-29 22:00:00,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [747f76ba] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:00:00,719 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-56) [20430170] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:00:00,720 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:00:00,721 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 11819362 >2014-04-29 22:00:00,736 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:00:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:00:02,257 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [e55af09] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4886d642 >2014-04-29 22:00:02,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [e55af09] Failed in GetHardwareInfoVDS method >2014-04-29 22:00:02,482 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [e55af09] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:00:02,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [e55af09] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@c5cac41 >2014-04-29 22:00:02,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [e55af09] HostName = 10.34.27.209 >2014-04-29 22:00:02,482 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [e55af09] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:00:02,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [e55af09] FINISH, GetHardwareInfoVDSCommand, log id: 4886d642 >2014-04-29 22:00:02,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [e55af09] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:00:02,495 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [e55af09] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:00:02,717 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-41) [12fed537] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:00:02,718 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-41) [56402d8d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:00:02,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-41) [56402d8d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7bf616e2 >2014-04-29 22:03:02,726 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-41) [56402d8d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:03:02,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-41) [56402d8d] FINISH, ConnectStorageServerVDSCommand, log id: 7bf616e2 >2014-04-29 22:03:02,727 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-41) [56402d8d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:03:02,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 272c679 >2014-04-29 22:03:03,841 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 272c679 >2014-04-29 22:03:03,841 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:03:03,854 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-41) [6f4bf4c9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:03:03,856 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [6f4bf4c9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3af05336 >2014-04-29 22:03:03,858 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [6f4bf4c9] FINISH, SetVdsStatusVDSCommand, log id: 3af05336 >2014-04-29 22:03:03,865 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [6f4bf4c9] Correlation ID: 6f4bf4c9, Job ID: 3b44053a-5c36-4348-8a48-a9dbd5ab9822, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:03:03,871 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [6f4bf4c9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:03:03,874 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [6f4bf4c9] Correlation ID: 12fed537, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:03:03,881 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-41) [838e908] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:03:03,883 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-41) [1ab00629] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:03:03,883 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) [1ab00629] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-95) [77f90510] Autorecovering 1 hosts >2014-04-29 22:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-95) [77f90510] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1c8d8eec >2014-04-29 22:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] FINISH, SetVdsStatusVDSCommand, log id: 1c8d8eec >2014-04-29 22:05:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2aafde30 >2014-04-29 22:05:01,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1f4b16d7 >2014-04-29 22:05:01,664 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] Failed in GetHardwareInfoVDS method >2014-04-29 22:05:01,664 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:05:01,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5273aee >2014-04-29 22:05:01,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] HostName = 10.34.27.209 >2014-04-29 22:05:01,664 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:05:01,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [739972c6] FINISH, GetHardwareInfoVDSCommand, log id: 1f4b16d7 >2014-04-29 22:05:01,669 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [739972c6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:05:01,678 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [13ded663] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:05:01,679 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [3fb7e461] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:05:01,680 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) [3fb7e461] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2aafde30 >2014-04-29 22:05:01,693 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [3fb7e461] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:05:01,694 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [3fb7e461] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:05:04,534 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 68d32a70 >2014-04-29 22:05:05,082 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] Failed in GetHardwareInfoVDS method >2014-04-29 22:05:05,083 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:05:05,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4b5b7d9c >2014-04-29 22:05:05,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] HostName = 10.34.27.209 >2014-04-29 22:05:05,083 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:05:05,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-25) [7d75a0be] FINISH, GetHardwareInfoVDSCommand, log id: 68d32a70 >2014-04-29 22:05:05,090 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [7d75a0be] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:05:05,101 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [7d75a0be] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:05:05,647 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-25) [5833407] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:05:05,649 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [1bf52b9f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:05:05,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [1bf52b9f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 326a5c71 >2014-04-29 22:08:05,657 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [1bf52b9f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:08:05,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-25) [1bf52b9f] FINISH, ConnectStorageServerVDSCommand, log id: 326a5c71 >2014-04-29 22:08:05,657 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-25) [1bf52b9f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:08:05,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 482023ae >2014-04-29 22:08:06,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 482023ae >2014-04-29 22:08:06,686 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:08:06,696 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-25) [220bc114] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:08:06,697 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [220bc114] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3e7c1056 >2014-04-29 22:08:06,700 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-25) [220bc114] FINISH, SetVdsStatusVDSCommand, log id: 3e7c1056 >2014-04-29 22:08:06,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [220bc114] Correlation ID: 220bc114, Job ID: c31a2515-51e5-44fe-a6bc-f41e65e5d568, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:08:06,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [220bc114] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:08:06,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-25) [220bc114] Correlation ID: 5833407, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:08:06,719 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-25) [a15c829] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:08:06,720 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-25) [91e52f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:08:06,721 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-25) [91e52f] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-94) [6ae8ff6] Autorecovering 1 hosts >2014-04-29 22:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-94) [6ae8ff6] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: bf5ea27 >2014-04-29 22:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] FINISH, SetVdsStatusVDSCommand, log id: bf5ea27 >2014-04-29 22:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1dc9fce6 >2014-04-29 22:10:00,849 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1296806e >2014-04-29 22:10:01,261 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] Failed in GetHardwareInfoVDS method >2014-04-29 22:10:01,262 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:10:01,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@572dbfc9 >2014-04-29 22:10:01,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] HostName = 10.34.27.209 >2014-04-29 22:10:01,262 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:10:01,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [311f58d2] FINISH, GetHardwareInfoVDSCommand, log id: 1296806e >2014-04-29 22:10:01,267 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [311f58d2] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:10:01,275 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-94) [45fc217d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:10:01,285 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-94) [6c6996dc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:10:01,286 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) [6c6996dc] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1dc9fce6 >2014-04-29 22:10:01,299 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [6c6996dc] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:10:01,300 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [6c6996dc] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:10:04,245 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [5223973b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2a8e4048 >2014-04-29 22:10:04,648 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [5223973b] Failed in GetHardwareInfoVDS method >2014-04-29 22:10:04,648 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [5223973b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:10:04,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [5223973b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@611e4ccb >2014-04-29 22:10:04,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [5223973b] HostName = 10.34.27.209 >2014-04-29 22:10:04,648 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [5223973b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:10:04,648 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [5223973b] FINISH, GetHardwareInfoVDSCommand, log id: 2a8e4048 >2014-04-29 22:10:04,654 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [5223973b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:10:04,673 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [5223973b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:10:05,082 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [740aad77] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:10:05,083 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [802e874] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:10:05,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [802e874] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 524eb2fe >2014-04-29 22:13:05,091 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [802e874] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:13:05,091 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [802e874] FINISH, ConnectStorageServerVDSCommand, log id: 524eb2fe >2014-04-29 22:13:05,091 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [802e874] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:13:05,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 35ab1a39 >2014-04-29 22:13:06,103 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 35ab1a39 >2014-04-29 22:13:06,103 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:13:06,218 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-81) [4e8d9e8d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:13:06,219 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [4e8d9e8d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4b941296 >2014-04-29 22:13:06,247 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [4e8d9e8d] FINISH, SetVdsStatusVDSCommand, log id: 4b941296 >2014-04-29 22:13:06,305 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [4e8d9e8d] Correlation ID: 4e8d9e8d, Job ID: 346500ac-84fc-4949-8d41-9de5f483059b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:13:06,369 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [4e8d9e8d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:13:06,389 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [4e8d9e8d] Correlation ID: 740aad77, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:13:06,448 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [424b4351] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:13:06,449 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [782db684] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:13:06,450 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [782db684] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [26e34cd9] Autorecovering 1 hosts >2014-04-29 22:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [26e34cd9] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4a43d633 >2014-04-29 22:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] FINISH, SetVdsStatusVDSCommand, log id: 4a43d633 >2014-04-29 22:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 18d673e4 >2014-04-29 22:15:01,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 465da637 >2014-04-29 22:15:01,592 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] Failed in GetHardwareInfoVDS method >2014-04-29 22:15:01,592 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:15:01,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3e15cc70 >2014-04-29 22:15:01,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] HostName = 10.34.27.209 >2014-04-29 22:15:01,592 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:15:01,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [9cea38a] FINISH, GetHardwareInfoVDSCommand, log id: 465da637 >2014-04-29 22:15:01,598 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [9cea38a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:15:01,609 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-65) [b1f9c5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:15:01,611 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-65) [2f82d73b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:15:01,612 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) [2f82d73b] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 18d673e4 >2014-04-29 22:15:01,628 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [2f82d73b] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:15:01,629 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [2f82d73b] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:15:04,090 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: c2ed966 >2014-04-29 22:15:04,604 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] Failed in GetHardwareInfoVDS method >2014-04-29 22:15:04,604 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:15:04,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@57c2bdc9 >2014-04-29 22:15:04,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] HostName = 10.34.27.209 >2014-04-29 22:15:04,604 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:15:04,604 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] FINISH, GetHardwareInfoVDSCommand, log id: c2ed966 >2014-04-29 22:15:04,610 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:15:04,621 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [5d8c58b7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:15:05,148 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-92) [47449e31] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:15:05,150 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-92) [71f3f6f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:15:05,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [71f3f6f8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7f9ffc9b >2014-04-29 22:18:05,163 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [71f3f6f8] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:18:05,163 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [71f3f6f8] FINISH, ConnectStorageServerVDSCommand, log id: 7f9ffc9b >2014-04-29 22:18:05,163 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-92) [71f3f6f8] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:18:05,167 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 61a684fc >2014-04-29 22:18:06,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 61a684fc >2014-04-29 22:18:06,230 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:18:06,241 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-92) [6894e8ee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:18:06,242 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [6894e8ee] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: c21fd32 >2014-04-29 22:18:06,244 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [6894e8ee] FINISH, SetVdsStatusVDSCommand, log id: c21fd32 >2014-04-29 22:18:06,250 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [6894e8ee] Correlation ID: 6894e8ee, Job ID: 76964b83-425f-479f-9636-ff1a867b0e7d, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:18:06,255 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [6894e8ee] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:18:06,257 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [6894e8ee] Correlation ID: 47449e31, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:18:06,261 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-92) [148f3467] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:18:06,262 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-92) [2346519] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:18:06,263 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [2346519] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) [6600b25d] Autorecovering 1 hosts >2014-04-29 22:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) [6600b25d] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 30ac1800 >2014-04-29 22:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] FINISH, SetVdsStatusVDSCommand, log id: 30ac1800 >2014-04-29 22:20:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 61a19471 >2014-04-29 22:20:01,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 501ebf1c >2014-04-29 22:20:01,596 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] Failed in GetHardwareInfoVDS method >2014-04-29 22:20:01,596 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:20:01,596 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7f3be44a >2014-04-29 22:20:01,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] HostName = 10.34.27.209 >2014-04-29 22:20:01,597 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:20:01,597 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [56b821ea] FINISH, GetHardwareInfoVDSCommand, log id: 501ebf1c >2014-04-29 22:20:01,603 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [56b821ea] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:20:01,613 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [4471b71a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:20:01,615 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [803988a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:20:01,616 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [803988a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 61a19471 >2014-04-29 22:20:01,631 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [803988a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:20:01,632 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [803988a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:20:03,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [72992f90] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 24f8a808 >2014-04-29 22:20:04,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [72992f90] Failed in GetHardwareInfoVDS method >2014-04-29 22:20:04,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [72992f90] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:20:04,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [72992f90] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@40dfb82e >2014-04-29 22:20:04,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [72992f90] HostName = 10.34.27.209 >2014-04-29 22:20:04,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [72992f90] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:20:04,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [72992f90] FINISH, GetHardwareInfoVDSCommand, log id: 24f8a808 >2014-04-29 22:20:04,482 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [72992f90] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:20:04,493 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [72992f90] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:20:05,049 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-31) [78568628] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:20:05,051 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [1bdca9b6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:20:05,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [1bdca9b6] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 40a8bd2e >2014-04-29 22:23:05,060 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [1bdca9b6] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:23:05,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [1bdca9b6] FINISH, ConnectStorageServerVDSCommand, log id: 40a8bd2e >2014-04-29 22:23:05,060 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [1bdca9b6] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:23:05,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 50c6a919 >2014-04-29 22:23:06,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 50c6a919 >2014-04-29 22:23:06,085 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:23:06,096 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-31) [7831f957] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:23:06,097 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [7831f957] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3ee18d5a >2014-04-29 22:23:06,103 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [7831f957] FINISH, SetVdsStatusVDSCommand, log id: 3ee18d5a >2014-04-29 22:23:06,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [7831f957] Correlation ID: 7831f957, Job ID: d57168a1-38b5-4df2-bd56-26a7a1dfa8fb, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:23:06,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [7831f957] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:23:06,117 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [7831f957] Correlation ID: 78568628, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:23:06,121 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-31) [6a7c9e4d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:23:06,122 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-31) [5f25aab3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:23:06,123 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [5f25aab3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-94) [6c6996dc] Autorecovering 1 hosts >2014-04-29 22:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-94) [6c6996dc] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6088e1ca >2014-04-29 22:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] FINISH, SetVdsStatusVDSCommand, log id: 6088e1ca >2014-04-29 22:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1256064c >2014-04-29 22:25:00,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4a4fe11b >2014-04-29 22:25:01,405 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] Failed in GetHardwareInfoVDS method >2014-04-29 22:25:01,405 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:25:01,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4243d9ba >2014-04-29 22:25:01,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] HostName = 10.34.27.209 >2014-04-29 22:25:01,405 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:25:01,405 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] FINISH, GetHardwareInfoVDSCommand, log id: 4a4fe11b >2014-04-29 22:25:01,411 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [3c1efaa4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:25:01,424 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-94) [3fa307be] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:25:01,426 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:25:01,427 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1256064c >2014-04-29 22:25:01,442 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:25:01,443 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:25:03,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [41c42308] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 21153da6 >2014-04-29 22:25:04,085 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [41c42308] Failed in GetHardwareInfoVDS method >2014-04-29 22:25:04,085 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [41c42308] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:25:04,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [41c42308] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@22ec6b8e >2014-04-29 22:25:04,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [41c42308] HostName = 10.34.27.209 >2014-04-29 22:25:04,085 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [41c42308] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:25:04,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [41c42308] FINISH, GetHardwareInfoVDSCommand, log id: 21153da6 >2014-04-29 22:25:04,089 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [41c42308] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:25:04,098 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [41c42308] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:25:04,512 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-14) [60460295] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:25:04,513 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [1ff14a03] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:25:04,521 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [1ff14a03] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4d828fef >2014-04-29 22:28:04,523 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [1ff14a03] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:28:04,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-14) [1ff14a03] FINISH, ConnectStorageServerVDSCommand, log id: 4d828fef >2014-04-29 22:28:04,523 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-14) [1ff14a03] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:28:04,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 34cfa938 >2014-04-29 22:28:05,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 34cfa938 >2014-04-29 22:28:05,651 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:28:05,662 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-14) [52d12e87] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:28:05,663 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [52d12e87] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7f28aac >2014-04-29 22:28:05,666 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [52d12e87] FINISH, SetVdsStatusVDSCommand, log id: 7f28aac >2014-04-29 22:28:05,672 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [52d12e87] Correlation ID: 52d12e87, Job ID: ab2081ee-61c5-4b0b-b1cd-2cc5e1dc93ae, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:28:05,678 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [52d12e87] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:28:05,680 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [52d12e87] Correlation ID: 60460295, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:28:05,684 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-14) [2ada9fdc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:28:05,685 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-14) [399037c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:28:05,686 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-14) [399037c0] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [582c9313] Autorecovering 1 hosts >2014-04-29 22:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [582c9313] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:30:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4b85a8a2 >2014-04-29 22:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] FINISH, SetVdsStatusVDSCommand, log id: 4b85a8a2 >2014-04-29 22:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6a35494a >2014-04-29 22:30:00,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 617a3dd5 >2014-04-29 22:30:00,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] Failed in GetHardwareInfoVDS method >2014-04-29 22:30:00,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:30:00,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@20719cf2 >2014-04-29 22:30:00,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] HostName = 10.34.27.209 >2014-04-29 22:30:00,967 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:30:00,967 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [60deda67] FINISH, GetHardwareInfoVDSCommand, log id: 617a3dd5 >2014-04-29 22:30:00,973 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [60deda67] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:30:00,982 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-33) [254f4f2d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:30:00,983 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-33) [14a6cef4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:30:00,984 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) [14a6cef4] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6a35494a >2014-04-29 22:30:00,999 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [14a6cef4] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:30:01,000 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [14a6cef4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:30:03,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [2346519] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1f82dd39 >2014-04-29 22:30:03,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [2346519] Failed in GetHardwareInfoVDS method >2014-04-29 22:30:03,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [2346519] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:30:03,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [2346519] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4889d958 >2014-04-29 22:30:03,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [2346519] HostName = 10.34.27.209 >2014-04-29 22:30:03,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [2346519] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:30:03,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [2346519] FINISH, GetHardwareInfoVDSCommand, log id: 1f82dd39 >2014-04-29 22:30:03,440 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [2346519] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:30:03,448 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [2346519] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:30:03,761 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-92) [6fd62468] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:30:03,763 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-92) [502b4ca0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:30:03,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [502b4ca0] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: ba78789 >2014-04-29 22:33:03,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [502b4ca0] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:33:03,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [502b4ca0] FINISH, ConnectStorageServerVDSCommand, log id: ba78789 >2014-04-29 22:33:03,771 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-92) [502b4ca0] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:33:03,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1a65082f >2014-04-29 22:33:04,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 1a65082f >2014-04-29 22:33:04,769 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:33:04,780 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-92) [c4ba8cb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:33:04,781 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [c4ba8cb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: cbee91d >2014-04-29 22:33:04,783 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [c4ba8cb] FINISH, SetVdsStatusVDSCommand, log id: cbee91d >2014-04-29 22:33:04,789 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [c4ba8cb] Correlation ID: c4ba8cb, Job ID: 47107e0e-bf54-4598-afa5-2e47e4317f0a, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:33:04,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [c4ba8cb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:33:04,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [c4ba8cb] Correlation ID: 6fd62468, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:33:04,805 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-92) [3f30c398] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:33:04,806 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-92) [7e9eb086] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:33:04,806 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [7e9eb086] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-62) [383fdecb] Autorecovering 1 hosts >2014-04-29 22:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-62) [383fdecb] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 35d1de63 >2014-04-29 22:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] FINISH, SetVdsStatusVDSCommand, log id: 35d1de63 >2014-04-29 22:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2653833b >2014-04-29 22:35:01,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 13b71a16 >2014-04-29 22:35:01,642 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] Failed in GetHardwareInfoVDS method >2014-04-29 22:35:01,642 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:35:01,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3a51a6f0 >2014-04-29 22:35:01,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] HostName = 10.34.27.209 >2014-04-29 22:35:01,643 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:35:01,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [45c2ac20] FINISH, GetHardwareInfoVDSCommand, log id: 13b71a16 >2014-04-29 22:35:01,648 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [45c2ac20] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:35:01,657 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-62) [65ffaa89] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:35:01,658 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-62) [39584f9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:35:01,659 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-62) [39584f9] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2653833b >2014-04-29 22:35:01,674 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [39584f9] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:35:01,675 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [39584f9] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:35:02,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 777a9162 >2014-04-29 22:35:02,893 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] Failed in GetHardwareInfoVDS method >2014-04-29 22:35:02,893 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:35:02,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@fdef636 >2014-04-29 22:35:02,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] HostName = 10.34.27.209 >2014-04-29 22:35:02,893 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:35:02,893 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [11ff5ead] FINISH, GetHardwareInfoVDSCommand, log id: 777a9162 >2014-04-29 22:35:02,898 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [11ff5ead] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:35:02,908 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [11ff5ead] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:35:03,415 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-73) [315cbf53] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:35:03,417 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [4952f5b5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:35:03,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [4952f5b5] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2947d0a1 >2014-04-29 22:36:05,008 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-83) [5473ac4a] Start refreshing all users data >2014-04-29 22:36:07,280 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [49295470] Attempting to update VMs/Templates Ovf. >2014-04-29 22:36:07,281 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [49295470] Attempting to update VM OVFs in Data Center NFS >2014-04-29 22:36:07,282 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [49295470] Successfully updated VM OVFs in Data Center NFS >2014-04-29 22:36:07,282 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [49295470] Attempting to update template OVFs in Data Center NFS >2014-04-29 22:36:07,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [49295470] Successfully updated templates OVFs in Data Center NFS >2014-04-29 22:36:07,283 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [49295470] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 22:36:07,284 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-20) [49295470] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 22:38:03,426 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [4952f5b5] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:38:03,426 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [4952f5b5] FINISH, ConnectStorageServerVDSCommand, log id: 2947d0a1 >2014-04-29 22:38:03,426 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [4952f5b5] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:38:03,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 6d6aea46 >2014-04-29 22:38:04,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 6d6aea46 >2014-04-29 22:38:04,482 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:38:04,493 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-73) [2832ed39] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:38:04,494 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [2832ed39] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4a32b142 >2014-04-29 22:38:04,496 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [2832ed39] FINISH, SetVdsStatusVDSCommand, log id: 4a32b142 >2014-04-29 22:38:04,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [2832ed39] Correlation ID: 2832ed39, Job ID: be351265-e2f8-4384-8fec-45b29eaaff0c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:38:04,508 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [2832ed39] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:38:04,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [2832ed39] Correlation ID: 315cbf53, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:38:04,514 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [7ef81db5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:38:04,515 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [334e6013] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:38:04,516 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) [334e6013] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) [3da45a1c] Autorecovering 1 hosts >2014-04-29 22:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) [3da45a1c] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3475afe1 >2014-04-29 22:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] FINISH, SetVdsStatusVDSCommand, log id: 3475afe1 >2014-04-29 22:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 12520cbd >2014-04-29 22:40:01,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 648c6d09 >2014-04-29 22:40:01,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] Failed in GetHardwareInfoVDS method >2014-04-29 22:40:01,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:40:01,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@16def066 >2014-04-29 22:40:01,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] HostName = 10.34.27.209 >2014-04-29 22:40:01,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:40:01,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [140bdb3f] FINISH, GetHardwareInfoVDSCommand, log id: 648c6d09 >2014-04-29 22:40:01,790 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [140bdb3f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:40:01,800 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [7b9ba21] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:40:01,801 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:40:01,801 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 12520cbd >2014-04-29 22:40:01,816 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:40:01,817 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:40:04,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 61419a95 >2014-04-29 22:40:05,499 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] Failed in GetHardwareInfoVDS method >2014-04-29 22:40:05,499 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:40:05,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1a0bde15 >2014-04-29 22:40:05,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] HostName = 10.34.27.209 >2014-04-29 22:40:05,499 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:40:05,499 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [42f3740b] FINISH, GetHardwareInfoVDSCommand, log id: 61419a95 >2014-04-29 22:40:05,505 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [42f3740b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:40:05,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [42f3740b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:40:06,035 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [1da7b15d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:40:06,037 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [3015626d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:40:06,043 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [3015626d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 31bdcd4e >2014-04-29 22:43:06,047 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [3015626d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:43:06,047 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [3015626d] FINISH, ConnectStorageServerVDSCommand, log id: 31bdcd4e >2014-04-29 22:43:06,047 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [3015626d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:43:06,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 526c5866 >2014-04-29 22:43:07,075 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 526c5866 >2014-04-29 22:43:07,075 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:43:07,120 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-84) [5707bab0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:43:07,121 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [5707bab0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4fed224e >2014-04-29 22:43:07,131 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [5707bab0] FINISH, SetVdsStatusVDSCommand, log id: 4fed224e >2014-04-29 22:43:07,137 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [5707bab0] Correlation ID: 5707bab0, Job ID: 8c8cd61c-432f-472c-a410-ebbb0a822046, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:43:07,144 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [5707bab0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:43:07,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [5707bab0] Correlation ID: 1da7b15d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:43:07,150 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [7b044801] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:43:07,151 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [3d34cc89] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:43:07,151 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) [3d34cc89] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-1) [730c095] Autorecovering 1 hosts >2014-04-29 22:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-1) [730c095] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:45:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 41d69b25 >2014-04-29 22:45:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] FINISH, SetVdsStatusVDSCommand, log id: 41d69b25 >2014-04-29 22:45:00,018 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7c371f8a >2014-04-29 22:45:01,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4af3b191 >2014-04-29 22:45:01,654 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] Failed in GetHardwareInfoVDS method >2014-04-29 22:45:01,654 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:45:01,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4b77edd4 >2014-04-29 22:45:01,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] HostName = 10.34.27.209 >2014-04-29 22:45:01,654 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:45:01,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-1) [20858feb] FINISH, GetHardwareInfoVDSCommand, log id: 4af3b191 >2014-04-29 22:45:01,661 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-1) [20858feb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:45:01,672 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-1) [3591b8b3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:45:01,673 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-1) [5bb17e35] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:45:01,674 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-1) [5bb17e35] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7c371f8a >2014-04-29 22:45:01,690 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-1) [5bb17e35] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:45:01,691 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-1) [5bb17e35] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:45:04,856 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3dcb14d7 >2014-04-29 22:45:05,393 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] Failed in GetHardwareInfoVDS method >2014-04-29 22:45:05,393 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:45:05,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5fc54ef2 >2014-04-29 22:45:05,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] HostName = 10.34.27.209 >2014-04-29 22:45:05,393 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:45:05,393 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] FINISH, GetHardwareInfoVDSCommand, log id: 3dcb14d7 >2014-04-29 22:45:05,399 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:45:05,411 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [6b57f1f3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:45:05,913 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-93) [1f82f16a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:45:05,915 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [6cf1ff08] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:45:05,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [6cf1ff08] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: bba6a3a >2014-04-29 22:48:05,923 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [6cf1ff08] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:48:05,923 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [6cf1ff08] FINISH, ConnectStorageServerVDSCommand, log id: bba6a3a >2014-04-29 22:48:05,923 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [6cf1ff08] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:48:05,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 78e558d7 >2014-04-29 22:48:06,904 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 78e558d7 >2014-04-29 22:48:06,905 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:48:06,916 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-93) [3e1b372d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:48:06,917 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [3e1b372d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6faa8509 >2014-04-29 22:48:06,920 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [3e1b372d] FINISH, SetVdsStatusVDSCommand, log id: 6faa8509 >2014-04-29 22:48:06,926 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [3e1b372d] Correlation ID: 3e1b372d, Job ID: d4cdd01f-578a-4bc5-8eca-87dae869403a, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:48:06,932 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [3e1b372d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:48:06,934 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [3e1b372d] Correlation ID: 1f82f16a, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:48:06,939 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [12e95036] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:48:06,941 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [4e428ec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:48:06,942 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4e428ec] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-98) [4f1248ed] Autorecovering 1 hosts >2014-04-29 22:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-98) [4f1248ed] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:50:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:50:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:50:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:50:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: ba222ce >2014-04-29 22:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] FINISH, SetVdsStatusVDSCommand, log id: ba222ce >2014-04-29 22:50:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 21275472 >2014-04-29 22:50:00,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2a00bc4c >2014-04-29 22:50:00,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] Failed in GetHardwareInfoVDS method >2014-04-29 22:50:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:50:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@164a371f >2014-04-29 22:50:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] HostName = 10.34.27.209 >2014-04-29 22:50:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:50:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4bea422c] FINISH, GetHardwareInfoVDSCommand, log id: 2a00bc4c >2014-04-29 22:50:00,708 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [4bea422c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:50:00,718 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-98) [64062df9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:50:00,719 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:50:00,720 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 21275472 >2014-04-29 22:50:00,734 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:50:00,735 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:50:01,299 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6aaa3bc4 >2014-04-29 22:50:01,522 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] Failed in GetHardwareInfoVDS method >2014-04-29 22:50:01,522 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:50:01,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4e39f791 >2014-04-29 22:50:01,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] HostName = 10.34.27.209 >2014-04-29 22:50:01,523 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:50:01,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [620b7ad7] FINISH, GetHardwareInfoVDSCommand, log id: 6aaa3bc4 >2014-04-29 22:50:01,535 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [620b7ad7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:50:01,576 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [620b7ad7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:50:01,799 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-56) [4c4aa439] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:50:01,800 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-56) [54c5fdd8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:50:01,807 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [54c5fdd8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6d216c7d >2014-04-29 22:53:01,809 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [54c5fdd8] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:53:01,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-56) [54c5fdd8] FINISH, ConnectStorageServerVDSCommand, log id: 6d216c7d >2014-04-29 22:53:01,809 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-56) [54c5fdd8] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:53:01,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 52e01342 >2014-04-29 22:53:02,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 52e01342 >2014-04-29 22:53:02,263 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:53:02,274 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-56) [11c8057f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:53:02,276 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [11c8057f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1901225f >2014-04-29 22:53:02,278 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [11c8057f] FINISH, SetVdsStatusVDSCommand, log id: 1901225f >2014-04-29 22:53:02,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [11c8057f] Correlation ID: 11c8057f, Job ID: baf85d45-be9a-4444-8770-274339460d2e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:53:02,290 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [11c8057f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:53:02,292 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [11c8057f] Correlation ID: 4c4aa439, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:53:02,296 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-56) [5ef70064] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:53:02,297 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-56) [42ccfee2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:53:02,297 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-56) [42ccfee2] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 22:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-53) [2a286eea] Autorecovering 1 hosts >2014-04-29 22:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-53) [2a286eea] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 22:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 22:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 22:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6978424a >2014-04-29 22:55:00,031 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] FINISH, SetVdsStatusVDSCommand, log id: 6978424a >2014-04-29 22:55:00,032 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5a320b83 >2014-04-29 22:55:00,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7765b250 >2014-04-29 22:55:00,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] Failed in GetHardwareInfoVDS method >2014-04-29 22:55:00,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:55:00,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@359ddf6d >2014-04-29 22:55:00,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] HostName = 10.34.27.209 >2014-04-29 22:55:00,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:55:00,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [4952bb58] FINISH, GetHardwareInfoVDSCommand, log id: 7765b250 >2014-04-29 22:55:00,723 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [4952bb58] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:55:00,733 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-53) [138099ae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:55:00,735 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:55:00,736 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5a320b83 >2014-04-29 22:55:00,750 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 22:55:00,752 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 22:55:02,650 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [420ef9d5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3b471fce >2014-04-29 22:55:02,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [420ef9d5] Failed in GetHardwareInfoVDS method >2014-04-29 22:55:02,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [420ef9d5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:55:02,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [420ef9d5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3a792bc9 >2014-04-29 22:55:02,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [420ef9d5] HostName = 10.34.27.209 >2014-04-29 22:55:02,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [420ef9d5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 22:55:02,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [420ef9d5] FINISH, GetHardwareInfoVDSCommand, log id: 3b471fce >2014-04-29 22:55:02,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [420ef9d5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 22:55:02,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [420ef9d5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 22:55:03,107 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-69) [355cbb1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:55:03,109 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [7fd35f7] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 22:55:03,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [7fd35f7] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 46b4bc28 >2014-04-29 22:58:03,118 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [7fd35f7] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 22:58:03,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [7fd35f7] FINISH, ConnectStorageServerVDSCommand, log id: 46b4bc28 >2014-04-29 22:58:03,118 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [7fd35f7] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 22:58:03,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4677bd94 >2014-04-29 22:58:03,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 4677bd94 >2014-04-29 22:58:03,578 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 22:58:03,590 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-69) [5e7df833] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:58:03,591 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [5e7df833] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 10f80941 >2014-04-29 22:58:03,593 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [5e7df833] FINISH, SetVdsStatusVDSCommand, log id: 10f80941 >2014-04-29 22:58:03,599 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [5e7df833] Correlation ID: 5e7df833, Job ID: 5eea54c2-4d4e-4796-adf0-6355e6ee123d, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 22:58:03,606 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [5e7df833] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 22:58:03,608 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [5e7df833] Correlation ID: 355cbb1, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 22:58:03,612 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-69) [67ebf776] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:58:03,613 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-69) [7e680b09] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 22:58:03,614 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [7e680b09] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) [42cb5d24] Autorecovering 1 hosts >2014-04-29 23:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) [42cb5d24] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2c18b5f1 >2014-04-29 23:00:00,025 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] FINISH, SetVdsStatusVDSCommand, log id: 2c18b5f1 >2014-04-29 23:00:00,026 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 64e1f7c7 >2014-04-29 23:00:00,529 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1a13961d >2014-04-29 23:00:00,769 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] Failed in GetHardwareInfoVDS method >2014-04-29 23:00:00,769 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:00:00,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7be4574a >2014-04-29 23:00:00,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] HostName = 10.34.27.209 >2014-04-29 23:00:00,769 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:00:00,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [33d42fb7] FINISH, GetHardwareInfoVDSCommand, log id: 1a13961d >2014-04-29 23:00:00,774 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [33d42fb7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:00:00,784 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-26) [6f10ecf8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:00:00,786 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-26) [722b448d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:00:00,786 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) [722b448d] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 64e1f7c7 >2014-04-29 23:00:00,803 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [722b448d] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:00:00,804 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [722b448d] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:00:04,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [49295470] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7de5484b >2014-04-29 23:00:04,484 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [49295470] Failed in GetHardwareInfoVDS method >2014-04-29 23:00:04,484 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [49295470] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:00:04,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [49295470] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2b96b004 >2014-04-29 23:00:04,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [49295470] HostName = 10.34.27.209 >2014-04-29 23:00:04,484 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [49295470] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:00:04,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [49295470] FINISH, GetHardwareInfoVDSCommand, log id: 7de5484b >2014-04-29 23:00:04,489 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [49295470] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:00:04,503 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [49295470] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:00:04,736 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-20) [11ead9c1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:00:04,737 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [2a8d9238] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:00:04,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [2a8d9238] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5e83ff99 >2014-04-29 23:03:04,746 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [2a8d9238] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:03:04,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-20) [2a8d9238] FINISH, ConnectStorageServerVDSCommand, log id: 5e83ff99 >2014-04-29 23:03:04,746 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-20) [2a8d9238] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:03:04,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 21b865ac >2014-04-29 23:03:05,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 21b865ac >2014-04-29 23:03:05,199 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:03:05,215 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-20) [3bc1e8a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:03:05,216 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [3bc1e8a6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: c00a83 >2014-04-29 23:03:05,218 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [3bc1e8a6] FINISH, SetVdsStatusVDSCommand, log id: c00a83 >2014-04-29 23:03:05,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [3bc1e8a6] Correlation ID: 3bc1e8a6, Job ID: 34cb5fa5-bd48-49cd-8b89-444b290c29a4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:03:05,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [3bc1e8a6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:03:05,233 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [3bc1e8a6] Correlation ID: 11ead9c1, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:03:05,238 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [76646608] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:03:05,240 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [4b3610c4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:03:05,240 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-20) [4b3610c4] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) [535824f4] Autorecovering 1 hosts >2014-04-29 23:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) [535824f4] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:05:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:05:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:05:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7975bdc4 >2014-04-29 23:05:00,025 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] FINISH, SetVdsStatusVDSCommand, log id: 7975bdc4 >2014-04-29 23:05:00,026 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7e783d50 >2014-04-29 23:05:00,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 412c4268 >2014-04-29 23:05:00,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] Failed in GetHardwareInfoVDS method >2014-04-29 23:05:00,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:05:00,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@40bbdb45 >2014-04-29 23:05:00,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] HostName = 10.34.27.209 >2014-04-29 23:05:00,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:05:00,713 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [3f72644] FINISH, GetHardwareInfoVDSCommand, log id: 412c4268 >2014-04-29 23:05:00,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [3f72644] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:05:00,728 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-51) [216ccebb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:05:00,734 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-51) [2d164364] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:05:00,735 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) [2d164364] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7e783d50 >2014-04-29 23:05:00,751 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [2d164364] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:05:00,752 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [2d164364] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:05:02,591 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 180858f >2014-04-29 23:05:02,816 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] Failed in GetHardwareInfoVDS method >2014-04-29 23:05:02,816 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:05:02,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1cc0700e >2014-04-29 23:05:02,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] HostName = 10.34.27.209 >2014-04-29 23:05:02,816 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:05:02,816 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [37a20f43] FINISH, GetHardwareInfoVDSCommand, log id: 180858f >2014-04-29 23:05:02,821 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [37a20f43] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:05:02,829 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [37a20f43] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:05:03,052 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-94) [3a8c1ac7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:05:03,053 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-94) [4fb515f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:05:03,060 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-94) [4fb515f8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 732ff8b6 >2014-04-29 23:08:03,062 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-94) [4fb515f8] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:08:03,062 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-94) [4fb515f8] FINISH, ConnectStorageServerVDSCommand, log id: 732ff8b6 >2014-04-29 23:08:03,062 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-94) [4fb515f8] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:08:03,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 2fc230e7 >2014-04-29 23:08:03,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 2fc230e7 >2014-04-29 23:08:03,506 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:08:03,516 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-94) [2580f630] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:08:03,517 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [2580f630] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3cdf7c5a >2014-04-29 23:08:03,519 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [2580f630] FINISH, SetVdsStatusVDSCommand, log id: 3cdf7c5a >2014-04-29 23:08:03,525 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [2580f630] Correlation ID: 2580f630, Job ID: 18ffe040-d628-484b-b791-16e1f78a74bf, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:08:03,531 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [2580f630] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:08:03,533 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [2580f630] Correlation ID: 3a8c1ac7, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:08:03,537 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-94) [1bf12aad] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:08:03,538 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-94) [4eb2e966] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:08:03,538 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [4eb2e966] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-34) [20b6ccf4] Autorecovering 1 hosts >2014-04-29 23:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-34) [20b6ccf4] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 41f87a1e >2014-04-29 23:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] FINISH, SetVdsStatusVDSCommand, log id: 41f87a1e >2014-04-29 23:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2486bd22 >2014-04-29 23:10:00,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2881b18e >2014-04-29 23:10:00,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] Failed in GetHardwareInfoVDS method >2014-04-29 23:10:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:10:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5aaff668 >2014-04-29 23:10:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] HostName = 10.34.27.209 >2014-04-29 23:10:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:10:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-34) [7108e2ab] FINISH, GetHardwareInfoVDSCommand, log id: 2881b18e >2014-04-29 23:10:00,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-34) [7108e2ab] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:10:00,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-34) [165b429a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:10:00,730 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-34) [773b1e9d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:10:00,730 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-34) [773b1e9d] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2486bd22 >2014-04-29 23:10:00,744 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [773b1e9d] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:10:00,744 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-34) [773b1e9d] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:10:03,891 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [21efa807] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7ee01115 >2014-04-29 23:10:04,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [21efa807] Failed in GetHardwareInfoVDS method >2014-04-29 23:10:04,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [21efa807] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:10:04,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [21efa807] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2881bac6 >2014-04-29 23:10:04,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [21efa807] HostName = 10.34.27.209 >2014-04-29 23:10:04,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [21efa807] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:10:04,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [21efa807] FINISH, GetHardwareInfoVDSCommand, log id: 7ee01115 >2014-04-29 23:10:04,119 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [21efa807] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:10:04,127 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [21efa807] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:10:04,349 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-80) [208f0d27] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:10:04,351 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [7f49d971] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:10:04,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [7f49d971] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3af9ee0a >2014-04-29 23:13:04,359 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [7f49d971] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:13:04,359 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [7f49d971] FINISH, ConnectStorageServerVDSCommand, log id: 3af9ee0a >2014-04-29 23:13:04,359 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [7f49d971] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:13:04,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 744c0523 >2014-04-29 23:13:04,809 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 744c0523 >2014-04-29 23:13:04,809 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:13:04,820 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-80) [560fcbdf] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:13:04,821 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [560fcbdf] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 337031d3 >2014-04-29 23:13:04,823 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [560fcbdf] FINISH, SetVdsStatusVDSCommand, log id: 337031d3 >2014-04-29 23:13:04,829 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [560fcbdf] Correlation ID: 560fcbdf, Job ID: 588d6a44-4603-41d6-a892-e35f3fd01e44, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:13:04,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [560fcbdf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:13:04,837 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [560fcbdf] Correlation ID: 208f0d27, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:13:04,841 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-80) [4310cfa7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:13:04,842 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-80) [10535fa3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:13:04,843 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) [10535fa3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-73) [334e6013] Autorecovering 1 hosts >2014-04-29 23:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-73) [334e6013] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 74d903b8 >2014-04-29 23:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] FINISH, SetVdsStatusVDSCommand, log id: 74d903b8 >2014-04-29 23:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 606c79c7 >2014-04-29 23:15:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 35fa9e49 >2014-04-29 23:15:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] Failed in GetHardwareInfoVDS method >2014-04-29 23:15:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:15:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@eb385e >2014-04-29 23:15:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] HostName = 10.34.27.209 >2014-04-29 23:15:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:15:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3025eff] FINISH, GetHardwareInfoVDSCommand, log id: 35fa9e49 >2014-04-29 23:15:00,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [3025eff] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:15:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [37ddff57] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:15:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [6a869533] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:15:00,717 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) [6a869533] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 606c79c7 >2014-04-29 23:15:00,732 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [6a869533] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:15:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [6a869533] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:15:02,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [52f20496] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: de15003 >2014-04-29 23:15:02,435 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [52f20496] Failed in GetHardwareInfoVDS method >2014-04-29 23:15:02,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [52f20496] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:15:02,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [52f20496] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@53335692 >2014-04-29 23:15:02,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [52f20496] HostName = 10.34.27.209 >2014-04-29 23:15:02,436 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [52f20496] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:15:02,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [52f20496] FINISH, GetHardwareInfoVDSCommand, log id: de15003 >2014-04-29 23:15:02,440 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [52f20496] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:15:02,449 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [52f20496] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:15:02,671 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [3708fa47] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:15:02,672 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [7a605d67] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:15:02,678 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [7a605d67] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 79d01a65 >2014-04-29 23:18:02,680 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [7a605d67] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:18:02,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [7a605d67] FINISH, ConnectStorageServerVDSCommand, log id: 79d01a65 >2014-04-29 23:18:02,680 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [7a605d67] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:18:02,683 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7733995a >2014-04-29 23:18:03,132 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 7733995a >2014-04-29 23:18:03,132 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:18:03,153 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-86) [5036f5af] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:18:03,155 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [5036f5af] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7d6ee65e >2014-04-29 23:18:03,162 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [5036f5af] FINISH, SetVdsStatusVDSCommand, log id: 7d6ee65e >2014-04-29 23:18:03,170 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [5036f5af] Correlation ID: 5036f5af, Job ID: d0f852ef-e2be-418f-96a9-b04b24d39d9c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:18:03,177 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [5036f5af] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:18:03,179 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [5036f5af] Correlation ID: 3708fa47, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:18:03,183 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [208e70de] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:18:03,184 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [2c8d8414] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:18:03,185 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [2c8d8414] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-75) [22863027] Autorecovering 1 hosts >2014-04-29 23:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-75) [22863027] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3199ff2b >2014-04-29 23:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] FINISH, SetVdsStatusVDSCommand, log id: 3199ff2b >2014-04-29 23:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 76372622 >2014-04-29 23:20:00,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 65a4cf31 >2014-04-29 23:20:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] Failed in GetHardwareInfoVDS method >2014-04-29 23:20:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:20:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@36a3e79d >2014-04-29 23:20:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] HostName = 10.34.27.209 >2014-04-29 23:20:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:20:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] FINISH, GetHardwareInfoVDSCommand, log id: 65a4cf31 >2014-04-29 23:20:00,707 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [4d5e0e4e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:20:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-75) [7484a078] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:20:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-75) [2f90db04] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:20:00,717 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) [2f90db04] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 76372622 >2014-04-29 23:20:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [2f90db04] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:20:00,732 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [2f90db04] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:20:03,551 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [58040b92] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7705678 >2014-04-29 23:20:03,790 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [58040b92] Failed in GetHardwareInfoVDS method >2014-04-29 23:20:03,790 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [58040b92] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:20:03,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [58040b92] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@603ce418 >2014-04-29 23:20:03,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [58040b92] HostName = 10.34.27.209 >2014-04-29 23:20:03,791 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [58040b92] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:20:03,791 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [58040b92] FINISH, GetHardwareInfoVDSCommand, log id: 7705678 >2014-04-29 23:20:03,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [58040b92] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:20:03,806 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [58040b92] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:20:04,041 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-12) [7e5ac892] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:20:04,043 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [54a5f524] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:20:04,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [54a5f524] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 23ee0d36 >2014-04-29 23:23:04,054 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [54a5f524] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:23:04,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [54a5f524] FINISH, ConnectStorageServerVDSCommand, log id: 23ee0d36 >2014-04-29 23:23:04,055 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [54a5f524] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:23:04,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7de8c3d5 >2014-04-29 23:23:04,539 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7de8c3d5 >2014-04-29 23:23:04,540 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:23:04,549 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [2b6accb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:23:04,550 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [2b6accb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7164338b >2014-04-29 23:23:04,552 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [2b6accb] FINISH, SetVdsStatusVDSCommand, log id: 7164338b >2014-04-29 23:23:04,558 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [2b6accb] Correlation ID: 2b6accb, Job ID: be43a08c-e064-4180-8810-e8056d590012, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:23:04,565 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [2b6accb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:23:04,567 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [2b6accb] Correlation ID: 7e5ac892, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:23:04,571 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [6f3fea8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:23:04,572 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:23:04,573 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-36) [7128c0f5] Autorecovering 1 hosts >2014-04-29 23:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-36) [7128c0f5] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 40aa0ea1 >2014-04-29 23:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] FINISH, SetVdsStatusVDSCommand, log id: 40aa0ea1 >2014-04-29 23:25:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: df5e016 >2014-04-29 23:25:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 70fcfa9f >2014-04-29 23:25:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] Failed in GetHardwareInfoVDS method >2014-04-29 23:25:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:25:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6c709659 >2014-04-29 23:25:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] HostName = 10.34.27.209 >2014-04-29 23:25:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:25:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [615cbbe7] FINISH, GetHardwareInfoVDSCommand, log id: 70fcfa9f >2014-04-29 23:25:00,735 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [615cbbe7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:25:00,796 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [1c12b967] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:25:00,804 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [7476700d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:25:00,806 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-36) [7476700d] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: df5e016 >2014-04-29 23:25:00,832 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [7476700d] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:25:00,833 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [7476700d] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:25:01,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 9cb09bf >2014-04-29 23:25:02,150 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] Failed in GetHardwareInfoVDS method >2014-04-29 23:25:02,150 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:25:02,150 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@70ecdf9e >2014-04-29 23:25:02,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] HostName = 10.34.27.209 >2014-04-29 23:25:02,151 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:25:02,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [10122b1e] FINISH, GetHardwareInfoVDSCommand, log id: 9cb09bf >2014-04-29 23:25:02,156 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [10122b1e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:25:02,164 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [10122b1e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:25:02,388 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-78) [652c5edb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:25:02,390 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [7b16fbdd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:25:02,402 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [7b16fbdd] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 697a4d36 >2014-04-29 23:28:02,403 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [7b16fbdd] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:28:02,403 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [7b16fbdd] FINISH, ConnectStorageServerVDSCommand, log id: 697a4d36 >2014-04-29 23:28:02,403 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [7b16fbdd] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:28:02,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 913dac3 >2014-04-29 23:28:02,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 913dac3 >2014-04-29 23:28:02,863 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:28:02,877 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-78) [5ccbefc6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:28:02,878 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [5ccbefc6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3aa7529d >2014-04-29 23:28:02,881 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [5ccbefc6] FINISH, SetVdsStatusVDSCommand, log id: 3aa7529d >2014-04-29 23:28:02,902 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [5ccbefc6] Correlation ID: 5ccbefc6, Job ID: d93f3b64-9ad2-4e5e-b43e-c8d75f66c37f, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:28:02,909 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [5ccbefc6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:28:02,912 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [5ccbefc6] Correlation ID: 652c5edb, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:28:02,916 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [7a19b5d1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:28:02,918 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [5b82faa2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:28:02,918 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [5b82faa2] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) [782db684] Autorecovering 1 hosts >2014-04-29 23:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) [782db684] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 75bc61a0 >2014-04-29 23:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] FINISH, SetVdsStatusVDSCommand, log id: 75bc61a0 >2014-04-29 23:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 33bbdd35 >2014-04-29 23:30:00,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 633619ac >2014-04-29 23:30:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] Failed in GetHardwareInfoVDS method >2014-04-29 23:30:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:30:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@65b6deb8 >2014-04-29 23:30:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] HostName = 10.34.27.209 >2014-04-29 23:30:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:30:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [563535ca] FINISH, GetHardwareInfoVDSCommand, log id: 633619ac >2014-04-29 23:30:00,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [563535ca] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:30:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [f63c56a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:30:00,718 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:30:00,719 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 33bbdd35 >2014-04-29 23:30:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:30:00,738 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:30:03,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1b71c228 >2014-04-29 23:30:03,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] Failed in GetHardwareInfoVDS method >2014-04-29 23:30:03,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:30:03,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6b1b95cc >2014-04-29 23:30:03,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] HostName = 10.34.27.209 >2014-04-29 23:30:03,498 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:30:03,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [7e11ee78] FINISH, GetHardwareInfoVDSCommand, log id: 1b71c228 >2014-04-29 23:30:03,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [7e11ee78] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:30:03,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [7e11ee78] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:30:03,731 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-30) [11dec296] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:30:03,733 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [75a89a53] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:30:03,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [75a89a53] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 66453a6a >2014-04-29 23:33:03,740 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [75a89a53] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:33:03,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [75a89a53] FINISH, ConnectStorageServerVDSCommand, log id: 66453a6a >2014-04-29 23:33:03,741 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [75a89a53] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:33:03,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 631159b5 >2014-04-29 23:33:04,198 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 631159b5 >2014-04-29 23:33:04,198 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:33:04,208 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-30) [2d76304b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:33:04,209 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [2d76304b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2f95cc59 >2014-04-29 23:33:04,211 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [2d76304b] FINISH, SetVdsStatusVDSCommand, log id: 2f95cc59 >2014-04-29 23:33:04,217 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [2d76304b] Correlation ID: 2d76304b, Job ID: ae675774-928e-43a6-a786-f44075ed0de1, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:33:04,223 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [2d76304b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:33:04,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [2d76304b] Correlation ID: 11dec296, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:33:04,229 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-30) [1549713d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:33:04,230 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:33:04,230 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [14a6cef4] Autorecovering 1 hosts >2014-04-29 23:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [14a6cef4] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2254d8d0 >2014-04-29 23:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] FINISH, SetVdsStatusVDSCommand, log id: 2254d8d0 >2014-04-29 23:35:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5c3f55fe >2014-04-29 23:35:00,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5de6697 >2014-04-29 23:35:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] Failed in GetHardwareInfoVDS method >2014-04-29 23:35:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:35:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4c214683 >2014-04-29 23:35:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] HostName = 10.34.27.209 >2014-04-29 23:35:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:35:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [6a126e19] FINISH, GetHardwareInfoVDSCommand, log id: 5de6697 >2014-04-29 23:35:00,716 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [6a126e19] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:35:00,732 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-33) [560d0a09] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:35:00,734 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-33) [470b0a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:35:00,735 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) [470b0a4] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5c3f55fe >2014-04-29 23:35:00,757 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [470b0a4] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:35:00,758 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [470b0a4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:35:01,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 335fea7c >2014-04-29 23:35:01,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] Failed in GetHardwareInfoVDS method >2014-04-29 23:35:01,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:35:01,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5e1b8d3d >2014-04-29 23:35:01,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] HostName = 10.34.27.209 >2014-04-29 23:35:01,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:35:01,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [fc07203] FINISH, GetHardwareInfoVDSCommand, log id: 335fea7c >2014-04-29 23:35:01,818 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [fc07203] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:35:01,827 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [fc07203] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:35:02,048 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-45) [66f13285] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:35:02,050 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [44f92ce3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:35:02,057 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [44f92ce3] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2a8a1595 >2014-04-29 23:36:05,010 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Start refreshing all users data >2014-04-29 23:36:07,284 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) [49478d71] Attempting to update VMs/Templates Ovf. >2014-04-29 23:36:07,285 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) [49478d71] Attempting to update VM OVFs in Data Center NFS >2014-04-29 23:36:07,286 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) [49478d71] Successfully updated VM OVFs in Data Center NFS >2014-04-29 23:36:07,286 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) [49478d71] Attempting to update template OVFs in Data Center NFS >2014-04-29 23:36:07,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) [49478d71] Successfully updated templates OVFs in Data Center NFS >2014-04-29 23:36:07,287 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) [49478d71] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-29 23:36:07,288 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-22) [49478d71] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-29 23:38:02,058 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [44f92ce3] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:38:02,058 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [44f92ce3] FINISH, ConnectStorageServerVDSCommand, log id: 2a8a1595 >2014-04-29 23:38:02,058 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [44f92ce3] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:38:02,061 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7fc1b7d5 >2014-04-29 23:38:02,533 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7fc1b7d5 >2014-04-29 23:38:02,533 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:38:02,543 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-45) [441c5d4f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:38:02,544 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [441c5d4f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5cb66b32 >2014-04-29 23:38:02,546 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [441c5d4f] FINISH, SetVdsStatusVDSCommand, log id: 5cb66b32 >2014-04-29 23:38:02,568 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [441c5d4f] Correlation ID: 441c5d4f, Job ID: 006ac120-550d-4baf-9fcf-834269107d52, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:38:02,574 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [441c5d4f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:38:02,576 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [441c5d4f] Correlation ID: 66f13285, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:38:02,580 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [5a863306] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:38:02,581 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [3d2b8079] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:38:02,581 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) [3d2b8079] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-73) [6a869533] Autorecovering 1 hosts >2014-04-29 23:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-73) [6a869533] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 624dbac8 >2014-04-29 23:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] FINISH, SetVdsStatusVDSCommand, log id: 624dbac8 >2014-04-29 23:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 444e88b3 >2014-04-29 23:40:00,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4c5f918a >2014-04-29 23:40:00,727 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] Failed in GetHardwareInfoVDS method >2014-04-29 23:40:00,727 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:40:00,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@164c2232 >2014-04-29 23:40:00,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] HostName = 10.34.27.209 >2014-04-29 23:40:00,727 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:40:00,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [41917b7a] FINISH, GetHardwareInfoVDSCommand, log id: 4c5f918a >2014-04-29 23:40:00,733 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [41917b7a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:40:00,749 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [4d7cdf42] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:40:00,756 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [227e7ef] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:40:00,757 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) [227e7ef] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 444e88b3 >2014-04-29 23:40:00,778 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [227e7ef] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:40:00,779 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [227e7ef] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:40:02,941 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 265cf1ef >2014-04-29 23:40:03,164 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] Failed in GetHardwareInfoVDS method >2014-04-29 23:40:03,164 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:40:03,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4217b970 >2014-04-29 23:40:03,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] HostName = 10.34.27.209 >2014-04-29 23:40:03,164 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:40:03,164 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [4c187ded] FINISH, GetHardwareInfoVDSCommand, log id: 265cf1ef >2014-04-29 23:40:03,169 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [4c187ded] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:40:03,177 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [4c187ded] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:40:03,400 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-98) [6603abca] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:40:03,402 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [3821987a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:40:03,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [3821987a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 11bed62 >2014-04-29 23:43:03,414 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [3821987a] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:43:03,414 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [3821987a] FINISH, ConnectStorageServerVDSCommand, log id: 11bed62 >2014-04-29 23:43:03,415 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [3821987a] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:43:03,418 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 47391f51 >2014-04-29 23:43:03,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 47391f51 >2014-04-29 23:43:03,869 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:43:03,881 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-98) [7bc68f05] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:43:03,883 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [7bc68f05] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2e211038 >2014-04-29 23:43:03,885 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [7bc68f05] FINISH, SetVdsStatusVDSCommand, log id: 2e211038 >2014-04-29 23:43:03,891 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [7bc68f05] Correlation ID: 7bc68f05, Job ID: 66c133a2-9e31-47e0-a54d-65326b96099d, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:43:03,898 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [7bc68f05] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:43:03,900 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [7bc68f05] Correlation ID: 6603abca, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:43:03,903 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-98) [576b0caf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:43:03,904 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-98) [21fb7024] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:43:03,905 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [21fb7024] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-57) [6627d833] Autorecovering 1 hosts >2014-04-29 23:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-57) [6627d833] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1bc5ddc5 >2014-04-29 23:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] FINISH, SetVdsStatusVDSCommand, log id: 1bc5ddc5 >2014-04-29 23:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 312452e1 >2014-04-29 23:45:00,488 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1f56226 >2014-04-29 23:45:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] Failed in GetHardwareInfoVDS method >2014-04-29 23:45:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:45:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@14feadcb >2014-04-29 23:45:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] HostName = 10.34.27.209 >2014-04-29 23:45:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:45:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [25ea3bec] FINISH, GetHardwareInfoVDSCommand, log id: 1f56226 >2014-04-29 23:45:00,724 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [25ea3bec] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:45:00,735 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [6ce76a17] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:45:00,736 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:45:00,736 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 312452e1 >2014-04-29 23:45:00,749 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:45:00,750 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:45:01,287 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6fb375b0 >2014-04-29 23:45:01,543 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] Failed in GetHardwareInfoVDS method >2014-04-29 23:45:01,544 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:45:01,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5cba6f5f >2014-04-29 23:45:01,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] HostName = 10.34.27.209 >2014-04-29 23:45:01,544 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:45:01,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-53) [574539e2] FINISH, GetHardwareInfoVDSCommand, log id: 6fb375b0 >2014-04-29 23:45:01,548 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [574539e2] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:45:01,556 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [574539e2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:45:01,809 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-53) [78b41ce4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:45:01,811 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-53) [2e208777] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:45:01,817 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [2e208777] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 263ff3f6 >2014-04-29 23:48:01,819 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [2e208777] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:48:01,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-53) [2e208777] FINISH, ConnectStorageServerVDSCommand, log id: 263ff3f6 >2014-04-29 23:48:01,819 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-53) [2e208777] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:48:01,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: dbd0c5a >2014-04-29 23:48:02,288 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: dbd0c5a >2014-04-29 23:48:02,288 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:48:02,299 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-53) [438756fe] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:48:02,300 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [438756fe] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4d3f0345 >2014-04-29 23:48:02,302 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-53) [438756fe] FINISH, SetVdsStatusVDSCommand, log id: 4d3f0345 >2014-04-29 23:48:02,308 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [438756fe] Correlation ID: 438756fe, Job ID: 99f94514-d031-4a2e-9507-01dda9e45a59, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:48:02,315 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [438756fe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:48:02,317 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-53) [438756fe] Correlation ID: 78b41ce4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:48:02,320 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-53) [7a71830d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:48:02,321 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-53) [3b1dbbaa] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:48:02,322 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-53) [3b1dbbaa] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) [722b448d] Autorecovering 1 hosts >2014-04-29 23:50:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) [722b448d] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:50:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:50:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:50:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:50:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 10277287 >2014-04-29 23:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] FINISH, SetVdsStatusVDSCommand, log id: 10277287 >2014-04-29 23:50:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 283cee2a >2014-04-29 23:50:00,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 22a0c68d >2014-04-29 23:50:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] Failed in GetHardwareInfoVDS method >2014-04-29 23:50:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:50:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@46337b44 >2014-04-29 23:50:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] HostName = 10.34.27.209 >2014-04-29 23:50:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:50:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7b0453de] FINISH, GetHardwareInfoVDSCommand, log id: 22a0c68d >2014-04-29 23:50:00,709 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [7b0453de] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:50:00,770 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-26) [3ea0a6d3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:50:00,771 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-26) [2c15a355] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:50:00,772 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) [2c15a355] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 283cee2a >2014-04-29 23:50:00,802 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [2c15a355] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:50:00,803 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [2c15a355] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:50:02,684 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1fea9efb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 60593642 >2014-04-29 23:50:02,925 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1fea9efb] Failed in GetHardwareInfoVDS method >2014-04-29 23:50:02,926 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1fea9efb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:50:02,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1fea9efb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@f95879 >2014-04-29 23:50:02,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1fea9efb] HostName = 10.34.27.209 >2014-04-29 23:50:02,926 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1fea9efb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:50:02,926 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-44) [1fea9efb] FINISH, GetHardwareInfoVDSCommand, log id: 60593642 >2014-04-29 23:50:02,930 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [1fea9efb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:50:02,938 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [1fea9efb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:50:03,161 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-44) [3696a0e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:50:03,162 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [403456b5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:50:03,169 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [403456b5] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: c0fd645 >2014-04-29 23:53:03,170 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [403456b5] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:53:03,170 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-44) [403456b5] FINISH, ConnectStorageServerVDSCommand, log id: c0fd645 >2014-04-29 23:53:03,170 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-44) [403456b5] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:53:03,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: f680d2c >2014-04-29 23:53:03,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: f680d2c >2014-04-29 23:53:03,614 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:53:03,627 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-44) [73b9e056] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:53:03,628 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [73b9e056] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: f995b9d >2014-04-29 23:53:03,630 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-44) [73b9e056] FINISH, SetVdsStatusVDSCommand, log id: f995b9d >2014-04-29 23:53:03,636 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [73b9e056] Correlation ID: 73b9e056, Job ID: ef92e412-b080-44f6-84c4-0f1e033d8b3c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:53:03,642 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [73b9e056] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:53:03,644 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-44) [73b9e056] Correlation ID: 3696a0e, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:53:03,648 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-44) [3e7a2abe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:53:03,649 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-44) [5b09a53a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:53:03,650 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-44) [5b09a53a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-29 23:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-63) [48b1d9c4] Autorecovering 1 hosts >2014-04-29 23:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-63) [48b1d9c4] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-29 23:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-29 23:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-29 23:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 13edf682 >2014-04-29 23:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] FINISH, SetVdsStatusVDSCommand, log id: 13edf682 >2014-04-29 23:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2255f1d6 >2014-04-29 23:55:00,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 29080fde >2014-04-29 23:55:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] Failed in GetHardwareInfoVDS method >2014-04-29 23:55:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:55:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@49c411c4 >2014-04-29 23:55:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] HostName = 10.34.27.209 >2014-04-29 23:55:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:55:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] FINISH, GetHardwareInfoVDSCommand, log id: 29080fde >2014-04-29 23:55:00,727 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [5e8f4e43] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:55:00,784 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-63) [717c3239] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:55:00,786 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:55:00,786 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2255f1d6 >2014-04-29 23:55:00,827 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-29 23:55:00,828 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-29 23:55:04,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2c9cf636 >2014-04-29 23:55:04,247 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] Failed in GetHardwareInfoVDS method >2014-04-29 23:55:04,247 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:55:04,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@74e86a1b >2014-04-29 23:55:04,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] HostName = 10.34.27.209 >2014-04-29 23:55:04,247 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-29 23:55:04,247 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [6963c53e] FINISH, GetHardwareInfoVDSCommand, log id: 2c9cf636 >2014-04-29 23:55:04,257 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [6963c53e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-29 23:55:04,269 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [6963c53e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-29 23:55:04,505 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-91) [194c39e6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:55:04,507 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [271d19fc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-29 23:55:04,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [271d19fc] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2627ed8e >2014-04-29 23:58:04,514 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [271d19fc] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-29 23:58:04,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [271d19fc] FINISH, ConnectStorageServerVDSCommand, log id: 2627ed8e >2014-04-29 23:58:04,514 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [271d19fc] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-29 23:58:04,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7f9ddeb5 >2014-04-29 23:58:04,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 7f9ddeb5 >2014-04-29 23:58:04,958 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-29 23:58:04,968 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-91) [69779fa0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:58:04,969 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) [69779fa0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 48bc2098 >2014-04-29 23:58:04,971 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) [69779fa0] FINISH, SetVdsStatusVDSCommand, log id: 48bc2098 >2014-04-29 23:58:04,977 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [69779fa0] Correlation ID: 69779fa0, Job ID: bc9123b2-ceb3-4f94-82c5-efa536a5a3a4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-29 23:58:04,983 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [69779fa0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-29 23:58:04,985 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [69779fa0] Correlation ID: 194c39e6, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-29 23:58:04,989 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [5c386930] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:58:04,990 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [2f6b8212] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-29 23:58:04,991 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) [2f6b8212] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:00:00,010 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-89) [1c8b0c67] Autorecovering 1 hosts >2014-04-30 00:00:00,010 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-89) [1c8b0c67] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:00:00,012 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:00:00,012 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:00:00,012 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:00:00,012 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:00:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5990654 >2014-04-30 00:00:00,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] FINISH, SetVdsStatusVDSCommand, log id: 5990654 >2014-04-30 00:00:00,020 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 322ba766 >2014-04-30 00:00:00,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 315185f5 >2014-04-30 00:00:00,758 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] Failed in GetHardwareInfoVDS method >2014-04-30 00:00:00,758 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:00:00,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@103c8bca >2014-04-30 00:00:00,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] HostName = 10.34.27.209 >2014-04-30 00:00:00,758 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:00:00,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [6466e3bf] FINISH, GetHardwareInfoVDSCommand, log id: 315185f5 >2014-04-30 00:00:00,763 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [6466e3bf] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:00:00,773 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [4120cfe4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:00:00,774 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [7b33c831] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:00:00,775 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) [7b33c831] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 322ba766 >2014-04-30 00:00:00,794 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [7b33c831] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:00:00,796 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [7b33c831] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:00:02,384 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 52cc7312 >2014-04-30 00:00:02,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] Failed in GetHardwareInfoVDS method >2014-04-30 00:00:02,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:00:02,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6c96cd35 >2014-04-30 00:00:02,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] HostName = 10.34.27.209 >2014-04-30 00:00:02,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:00:02,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1cb759e4] FINISH, GetHardwareInfoVDSCommand, log id: 52cc7312 >2014-04-30 00:00:02,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [1cb759e4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:00:02,703 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [1cb759e4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:00:02,985 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-48) [570f1b72] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:00:02,987 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [f4ae138] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:00:02,995 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [f4ae138] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2335d3ae >2014-04-30 00:03:02,997 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [f4ae138] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:03:02,997 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [f4ae138] FINISH, ConnectStorageServerVDSCommand, log id: 2335d3ae >2014-04-30 00:03:02,997 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [f4ae138] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:03:03,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7219864f >2014-04-30 00:03:03,455 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 7219864f >2014-04-30 00:03:03,455 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:03:03,467 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-48) [50fe201e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:03:03,468 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [50fe201e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7211af27 >2014-04-30 00:03:03,470 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [50fe201e] FINISH, SetVdsStatusVDSCommand, log id: 7211af27 >2014-04-30 00:03:03,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [50fe201e] Correlation ID: 50fe201e, Job ID: 8e3430fe-9a70-4555-b6a5-2a19487b9ad3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:03:03,483 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [50fe201e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:03:03,485 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [50fe201e] Correlation ID: 570f1b72, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:03:03,489 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [20dc2601] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:03:03,490 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [5dad83cf] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:03:03,491 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [5dad83cf] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [2253937a] Autorecovering 1 hosts >2014-04-30 00:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [2253937a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6b0f6bef >2014-04-30 00:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] FINISH, SetVdsStatusVDSCommand, log id: 6b0f6bef >2014-04-30 00:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1380b897 >2014-04-30 00:05:00,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4d7bf9b1 >2014-04-30 00:05:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] Failed in GetHardwareInfoVDS method >2014-04-30 00:05:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:05:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@645fb35 >2014-04-30 00:05:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] HostName = 10.34.27.209 >2014-04-30 00:05:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:05:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [3ed111a7] FINISH, GetHardwareInfoVDSCommand, log id: 4d7bf9b1 >2014-04-30 00:05:00,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [3ed111a7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:05:00,714 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [5491f283] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:05:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [2b8adcc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:05:00,716 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [2b8adcc] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1380b897 >2014-04-30 00:05:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [2b8adcc] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:05:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [2b8adcc] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:05:03,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1ab00629] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3457d369 >2014-04-30 00:05:04,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1ab00629] Failed in GetHardwareInfoVDS method >2014-04-30 00:05:04,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1ab00629] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:05:04,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1ab00629] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6cb6ff2d >2014-04-30 00:05:04,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1ab00629] HostName = 10.34.27.209 >2014-04-30 00:05:04,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1ab00629] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:05:04,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [1ab00629] FINISH, GetHardwareInfoVDSCommand, log id: 3457d369 >2014-04-30 00:05:04,071 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [1ab00629] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:05:04,080 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [1ab00629] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:05:04,302 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-41) [38abbbbb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:05:04,304 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-41) [2259a288] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:05:04,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-41) [2259a288] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 32bb5c02 >2014-04-30 00:08:04,313 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-41) [2259a288] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:08:04,313 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-41) [2259a288] FINISH, ConnectStorageServerVDSCommand, log id: 32bb5c02 >2014-04-30 00:08:04,313 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-41) [2259a288] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:08:04,318 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 26c0d5f1 >2014-04-30 00:08:04,776 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 26c0d5f1 >2014-04-30 00:08:04,776 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:08:04,787 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-41) [54d596d8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:08:04,788 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [54d596d8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 20901bad >2014-04-30 00:08:04,790 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [54d596d8] FINISH, SetVdsStatusVDSCommand, log id: 20901bad >2014-04-30 00:08:04,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [54d596d8] Correlation ID: 54d596d8, Job ID: d1df6c6a-5721-484d-a3b2-d5e8cb0b71ab, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:08:04,832 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [54d596d8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:08:04,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [54d596d8] Correlation ID: 38abbbbb, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:08:04,838 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-41) [87943f3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:08:04,839 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-41) [2fc678f0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:08:04,840 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-41) [2fc678f0] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-75) [2f90db04] Autorecovering 1 hosts >2014-04-30 00:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-75) [2f90db04] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 34a000ff >2014-04-30 00:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] FINISH, SetVdsStatusVDSCommand, log id: 34a000ff >2014-04-30 00:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 31b07866 >2014-04-30 00:10:00,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 13b9827d >2014-04-30 00:10:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] Failed in GetHardwareInfoVDS method >2014-04-30 00:10:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:10:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7825e6a8 >2014-04-30 00:10:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] HostName = 10.34.27.209 >2014-04-30 00:10:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:10:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-75) [54bb040a] FINISH, GetHardwareInfoVDSCommand, log id: 13b9827d >2014-04-30 00:10:00,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-75) [54bb040a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:10:00,892 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-75) [1484ff52] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:10:00,893 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-75) [3c2d0529] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:10:00,894 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-75) [3c2d0529] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 31b07866 >2014-04-30 00:10:00,909 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [3c2d0529] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:10:00,910 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-75) [3c2d0529] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:10:02,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 775ea3d >2014-04-30 00:10:02,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] Failed in GetHardwareInfoVDS method >2014-04-30 00:10:02,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:10:02,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4e654e9e >2014-04-30 00:10:02,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] HostName = 10.34.27.209 >2014-04-30 00:10:02,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:10:02,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [2126616e] FINISH, GetHardwareInfoVDSCommand, log id: 775ea3d >2014-04-30 00:10:02,429 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [2126616e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:10:02,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [2126616e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:10:02,658 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-57) [1c7c9e5e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:10:02,660 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [762ad764] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:10:02,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [762ad764] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1cc39830 >2014-04-30 00:13:02,667 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [762ad764] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:13:02,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [762ad764] FINISH, ConnectStorageServerVDSCommand, log id: 1cc39830 >2014-04-30 00:13:02,667 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [762ad764] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:13:02,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 463d0c7f >2014-04-30 00:13:03,110 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 463d0c7f >2014-04-30 00:13:03,110 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:13:03,120 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-57) [1e9c0380] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:13:03,121 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [1e9c0380] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4b148b30 >2014-04-30 00:13:03,123 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [1e9c0380] FINISH, SetVdsStatusVDSCommand, log id: 4b148b30 >2014-04-30 00:13:03,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [1e9c0380] Correlation ID: 1e9c0380, Job ID: e35d01e5-ef29-4494-8268-ff3f5aa8fac5, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:13:03,136 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [1e9c0380] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:13:03,138 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [1e9c0380] Correlation ID: 1c7c9e5e, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:13:03,141 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [57572aee] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:13:03,142 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [58a53c0f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:13:03,143 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [58a53c0f] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) [2c15a355] Autorecovering 1 hosts >2014-04-30 00:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-26) [2c15a355] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: fd2fd46 >2014-04-30 00:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] FINISH, SetVdsStatusVDSCommand, log id: fd2fd46 >2014-04-30 00:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 680db461 >2014-04-30 00:15:00,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 247a4f0c >2014-04-30 00:15:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] Failed in GetHardwareInfoVDS method >2014-04-30 00:15:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:15:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@59dbaf89 >2014-04-30 00:15:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] HostName = 10.34.27.209 >2014-04-30 00:15:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:15:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [7cd8784e] FINISH, GetHardwareInfoVDSCommand, log id: 247a4f0c >2014-04-30 00:15:00,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [7cd8784e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:15:00,893 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-26) [10d48a8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:15:00,894 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:15:00,895 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 680db461 >2014-04-30 00:15:00,910 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:15:00,911 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:15:03,511 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4a0bcc6b >2014-04-30 00:15:03,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] Failed in GetHardwareInfoVDS method >2014-04-30 00:15:03,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:15:03,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4c99aac7 >2014-04-30 00:15:03,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] HostName = 10.34.27.209 >2014-04-30 00:15:03,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:15:03,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [b425a46] FINISH, GetHardwareInfoVDSCommand, log id: 4a0bcc6b >2014-04-30 00:15:03,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [b425a46] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:15:03,751 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [b425a46] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:15:03,975 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-88) [29c072fd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:15:03,977 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-88) [291130f8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:15:03,983 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-88) [291130f8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c323586 >2014-04-30 00:18:03,985 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-88) [291130f8] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:18:03,985 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-88) [291130f8] FINISH, ConnectStorageServerVDSCommand, log id: 6c323586 >2014-04-30 00:18:03,985 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-88) [291130f8] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:18:03,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 2db270cc >2014-04-30 00:18:04,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 2db270cc >2014-04-30 00:18:04,457 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:18:04,470 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-88) [28b7f696] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:18:04,471 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) [28b7f696] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3848cc2 >2014-04-30 00:18:04,474 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) [28b7f696] FINISH, SetVdsStatusVDSCommand, log id: 3848cc2 >2014-04-30 00:18:04,480 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [28b7f696] Correlation ID: 28b7f696, Job ID: e72bdc86-b512-4a54-b8f0-4c12a34bed5b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:18:04,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [28b7f696] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:18:04,488 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [28b7f696] Correlation ID: 29c072fd, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:18:04,493 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-88) [2f53f3ca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:18:04,494 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-88) [6aba72d1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:18:04,494 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-88) [6aba72d1] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-17) [c5ba562] Autorecovering 1 hosts >2014-04-30 00:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-17) [c5ba562] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 42b5299 >2014-04-30 00:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] FINISH, SetVdsStatusVDSCommand, log id: 42b5299 >2014-04-30 00:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7b3bda74 >2014-04-30 00:20:00,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5e793fb3 >2014-04-30 00:20:00,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] Failed in GetHardwareInfoVDS method >2014-04-30 00:20:00,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:20:00,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6a453139 >2014-04-30 00:20:00,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] HostName = 10.34.27.209 >2014-04-30 00:20:00,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:20:00,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [3163b69b] FINISH, GetHardwareInfoVDSCommand, log id: 5e793fb3 >2014-04-30 00:20:00,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [3163b69b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:20:00,886 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-17) [244c9d45] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:20:00,887 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:20:00,888 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7b3bda74 >2014-04-30 00:20:00,904 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:20:00,908 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [7060185e] Correlation ID: 3163b69b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was autorecovered. >2014-04-30 00:20:00,908 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:20:01,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7bca9712 >2014-04-30 00:20:02,076 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] Failed in GetHardwareInfoVDS method >2014-04-30 00:20:02,076 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:20:02,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@31845e3c >2014-04-30 00:20:02,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] HostName = 10.34.27.209 >2014-04-30 00:20:02,076 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:20:02,076 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [3d03539b] FINISH, GetHardwareInfoVDSCommand, log id: 7bca9712 >2014-04-30 00:20:02,080 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [3d03539b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:20:02,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [3d03539b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:20:02,310 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-72) [6b5ef844] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:20:02,311 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [3c384e8b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:20:02,317 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [3c384e8b] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 247f4eff >2014-04-30 00:23:02,319 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [3c384e8b] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:23:02,319 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-72) [3c384e8b] FINISH, ConnectStorageServerVDSCommand, log id: 247f4eff >2014-04-30 00:23:02,319 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-72) [3c384e8b] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:23:02,324 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3aa6b6fd >2014-04-30 00:23:02,767 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-45) FINISH, ConnectStoragePoolVDSCommand, log id: 3aa6b6fd >2014-04-30 00:23:02,767 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-45) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:23:02,777 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-72) [3db561be] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:23:02,778 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [3db561be] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6cc56ff3 >2014-04-30 00:23:02,780 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [3db561be] FINISH, SetVdsStatusVDSCommand, log id: 6cc56ff3 >2014-04-30 00:23:02,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [3db561be] Correlation ID: 3db561be, Job ID: f2ca0325-0b69-4858-82aa-b25ce389d859, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:23:02,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [3db561be] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:23:02,794 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [3db561be] Correlation ID: 6b5ef844, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:23:02,797 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-72) [2e2eaa52] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:23:02,798 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-72) [58cc4101] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:23:02,799 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-72) [58cc4101] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-67) [21498e1e] Autorecovering 1 hosts >2014-04-30 00:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-67) [21498e1e] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3ca1196 >2014-04-30 00:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] FINISH, SetVdsStatusVDSCommand, log id: 3ca1196 >2014-04-30 00:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4db0597e >2014-04-30 00:25:00,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 42772bff >2014-04-30 00:25:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] Failed in GetHardwareInfoVDS method >2014-04-30 00:25:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:25:00,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1de99470 >2014-04-30 00:25:00,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] HostName = 10.34.27.209 >2014-04-30 00:25:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:25:00,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-67) [13ee4143] FINISH, GetHardwareInfoVDSCommand, log id: 42772bff >2014-04-30 00:25:00,874 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-67) [13ee4143] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:25:00,883 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-67) [7f7d9a78] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:25:00,884 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-67) [6f2a3568] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:25:00,884 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-67) [6f2a3568] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4db0597e >2014-04-30 00:25:00,898 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [6f2a3568] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:25:00,898 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-67) [6f2a3568] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:25:03,192 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7ba7c2a >2014-04-30 00:25:03,542 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] Failed in GetHardwareInfoVDS method >2014-04-30 00:25:03,542 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:25:03,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@642dad5c >2014-04-30 00:25:03,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] HostName = 10.34.27.209 >2014-04-30 00:25:03,542 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:25:03,542 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [796995ca] FINISH, GetHardwareInfoVDSCommand, log id: 7ba7c2a >2014-04-30 00:25:03,547 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [796995ca] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:25:03,585 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [796995ca] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:25:03,949 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-58) [3eebeec5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:25:03,951 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [405186d8] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:25:03,957 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [405186d8] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 251f24f3 >2014-04-30 00:28:03,959 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [405186d8] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:28:03,959 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-58) [405186d8] FINISH, ConnectStorageServerVDSCommand, log id: 251f24f3 >2014-04-30 00:28:03,959 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-58) [405186d8] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:28:03,963 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7a122a83 >2014-04-30 00:28:04,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7a122a83 >2014-04-30 00:28:04,407 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:28:04,419 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-58) [1326244e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:28:04,420 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [1326244e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5fb21391 >2014-04-30 00:28:04,423 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [1326244e] FINISH, SetVdsStatusVDSCommand, log id: 5fb21391 >2014-04-30 00:28:04,429 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [1326244e] Correlation ID: 1326244e, Job ID: fd18a2d5-9dfc-45b9-bdff-be3571eb334e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:28:04,437 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [1326244e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:28:04,439 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [1326244e] Correlation ID: 3eebeec5, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:28:04,443 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [44fcfd71] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:28:04,444 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [17ec17ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:28:04,445 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-58) [17ec17ae] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-31) [5f25aab3] Autorecovering 1 hosts >2014-04-30 00:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-31) [5f25aab3] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 26a4dafb >2014-04-30 00:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] FINISH, SetVdsStatusVDSCommand, log id: 26a4dafb >2014-04-30 00:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4cec9199 >2014-04-30 00:30:00,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: c8f6ef2 >2014-04-30 00:30:00,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] Failed in GetHardwareInfoVDS method >2014-04-30 00:30:00,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:30:00,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1cbcd84f >2014-04-30 00:30:00,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] HostName = 10.34.27.209 >2014-04-30 00:30:00,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:30:00,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [5e3a6703] FINISH, GetHardwareInfoVDSCommand, log id: c8f6ef2 >2014-04-30 00:30:00,872 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [5e3a6703] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:30:00,880 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-31) [34483243] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:30:00,881 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:30:00,882 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4cec9199 >2014-04-30 00:30:00,905 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:30:00,907 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:30:01,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [b9dcd39] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 38f87828 >2014-04-30 00:30:02,033 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [b9dcd39] Failed in GetHardwareInfoVDS method >2014-04-30 00:30:02,033 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [b9dcd39] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:30:02,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [b9dcd39] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4098a750 >2014-04-30 00:30:02,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [b9dcd39] HostName = 10.34.27.209 >2014-04-30 00:30:02,033 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [b9dcd39] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:30:02,034 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [b9dcd39] FINISH, GetHardwareInfoVDSCommand, log id: 38f87828 >2014-04-30 00:30:02,049 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [b9dcd39] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:30:02,058 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [b9dcd39] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:30:02,287 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-13) [5530d69f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:30:02,288 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-13) [675e4531] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:30:02,295 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [675e4531] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: b71ac53 >2014-04-30 00:33:02,296 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [675e4531] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:33:02,296 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [675e4531] FINISH, ConnectStorageServerVDSCommand, log id: b71ac53 >2014-04-30 00:33:02,296 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-13) [675e4531] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:33:02,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4d5be902 >2014-04-30 00:33:02,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 4d5be902 >2014-04-30 00:33:02,753 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:33:02,762 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-13) [71dec300] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:33:02,763 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [71dec300] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 71443e60 >2014-04-30 00:33:02,765 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [71dec300] FINISH, SetVdsStatusVDSCommand, log id: 71443e60 >2014-04-30 00:33:02,771 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [71dec300] Correlation ID: 71dec300, Job ID: 006007df-8867-4f9b-a051-91ae5d3f8972, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:33:02,777 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [71dec300] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:33:02,780 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [71dec300] Correlation ID: 5530d69f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:33:02,784 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-13) [44670fa2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:33:02,785 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:33:02,785 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-84) [3d34cc89] Autorecovering 1 hosts >2014-04-30 00:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-84) [3d34cc89] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 41197a79 >2014-04-30 00:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] FINISH, SetVdsStatusVDSCommand, log id: 41197a79 >2014-04-30 00:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6bf0d516 >2014-04-30 00:35:00,642 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 13957578 >2014-04-30 00:35:00,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] Failed in GetHardwareInfoVDS method >2014-04-30 00:35:00,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:35:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@14bcaf5b >2014-04-30 00:35:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] HostName = 10.34.27.209 >2014-04-30 00:35:00,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:35:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [10c1c5d] FINISH, GetHardwareInfoVDSCommand, log id: 13957578 >2014-04-30 00:35:00,870 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [10c1c5d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:35:00,884 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [2bb93ab4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:35:00,885 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:35:00,886 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6bf0d516 >2014-04-30 00:35:00,905 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:35:00,906 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:35:03,144 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7b86efe5 >2014-04-30 00:35:03,368 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Failed in GetHardwareInfoVDS method >2014-04-30 00:35:03,368 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:35:03,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@57efe2be >2014-04-30 00:35:03,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] HostName = 10.34.27.209 >2014-04-30 00:35:03,368 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:35:03,368 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] FINISH, GetHardwareInfoVDSCommand, log id: 7b86efe5 >2014-04-30 00:35:03,373 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:35:03,382 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [1c23fe6b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:35:03,604 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-68) [7fa53741] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:35:03,606 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [7852f655] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:35:03,614 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [7852f655] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3eb8baeb >2014-04-30 00:36:05,013 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-38) [1aa55f29] Start refreshing all users data >2014-04-30 00:36:07,288 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Attempting to update VMs/Templates Ovf. >2014-04-30 00:36:07,289 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Attempting to update VM OVFs in Data Center NFS >2014-04-30 00:36:07,290 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Successfully updated VM OVFs in Data Center NFS >2014-04-30 00:36:07,290 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Attempting to update template OVFs in Data Center NFS >2014-04-30 00:36:07,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Successfully updated templates OVFs in Data Center NFS >2014-04-30 00:36:07,291 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-30 00:36:07,292 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-30 00:38:03,618 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [7852f655] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:38:03,618 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [7852f655] FINISH, ConnectStorageServerVDSCommand, log id: 3eb8baeb >2014-04-30 00:38:03,618 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [7852f655] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:38:03,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 50941bce >2014-04-30 00:38:04,083 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 50941bce >2014-04-30 00:38:04,083 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:38:04,093 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-68) [23354d7d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:38:04,094 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [23354d7d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6b8167b1 >2014-04-30 00:38:04,096 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [23354d7d] FINISH, SetVdsStatusVDSCommand, log id: 6b8167b1 >2014-04-30 00:38:04,101 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [23354d7d] Correlation ID: 23354d7d, Job ID: 927f7366-0299-414a-9905-484848bd8e37, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:38:04,108 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [23354d7d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:38:04,110 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [23354d7d] Correlation ID: 7fa53741, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:38:04,114 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [4094a00f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:38:04,115 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [3e1ca74a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:38:04,116 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) [3e1ca74a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-95) [3fb7e461] Autorecovering 1 hosts >2014-04-30 00:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-95) [3fb7e461] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6d8ff110 >2014-04-30 00:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] FINISH, SetVdsStatusVDSCommand, log id: 6d8ff110 >2014-04-30 00:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 78b7f808 >2014-04-30 00:40:00,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6f7bc13b >2014-04-30 00:40:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] Failed in GetHardwareInfoVDS method >2014-04-30 00:40:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:40:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@25cb41ab >2014-04-30 00:40:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] HostName = 10.34.27.209 >2014-04-30 00:40:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:40:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] FINISH, GetHardwareInfoVDSCommand, log id: 6f7bc13b >2014-04-30 00:40:00,888 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [7cceb8ef] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:40:00,900 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [61d2578c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:40:00,901 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [3376995f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:40:00,902 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) [3376995f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 78b7f808 >2014-04-30 00:40:00,923 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [3376995f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:40:00,925 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [3376995f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:40:01,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [5b82faa2] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: dede913 >2014-04-30 00:40:01,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [5b82faa2] Failed in GetHardwareInfoVDS method >2014-04-30 00:40:01,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [5b82faa2] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:40:01,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [5b82faa2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7e60e589 >2014-04-30 00:40:01,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [5b82faa2] HostName = 10.34.27.209 >2014-04-30 00:40:01,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [5b82faa2] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:40:01,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [5b82faa2] FINISH, GetHardwareInfoVDSCommand, log id: dede913 >2014-04-30 00:40:01,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [5b82faa2] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:40:01,712 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [5b82faa2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:40:01,934 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-78) [4c36b2dc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:40:01,936 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [4f3ed132] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:40:01,942 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f3ed132] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5d43a5ac >2014-04-30 00:43:01,944 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f3ed132] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:43:01,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f3ed132] FINISH, ConnectStorageServerVDSCommand, log id: 5d43a5ac >2014-04-30 00:43:01,944 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [4f3ed132] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:43:01,948 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7d7df24f >2014-04-30 00:43:02,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7d7df24f >2014-04-30 00:43:02,390 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:43:02,400 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-78) [3994d465] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:43:02,401 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [3994d465] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4ff1cc49 >2014-04-30 00:43:02,403 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [3994d465] FINISH, SetVdsStatusVDSCommand, log id: 4ff1cc49 >2014-04-30 00:43:02,409 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [3994d465] Correlation ID: 3994d465, Job ID: 6dded69a-e025-455a-9ff8-7c86473ec318, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:43:02,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [3994d465] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:43:02,418 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [3994d465] Correlation ID: 4c36b2dc, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:43:02,423 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [4e2a7d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:43:02,424 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [1355ae52] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:43:02,425 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [1355ae52] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-93) [4e428ec] Autorecovering 1 hosts >2014-04-30 00:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-93) [4e428ec] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:45:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 585245a4 >2014-04-30 00:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] FINISH, SetVdsStatusVDSCommand, log id: 585245a4 >2014-04-30 00:45:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5732b74 >2014-04-30 00:45:00,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 670855a8 >2014-04-30 00:45:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] Failed in GetHardwareInfoVDS method >2014-04-30 00:45:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:45:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@542fbd70 >2014-04-30 00:45:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] HostName = 10.34.27.209 >2014-04-30 00:45:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:45:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [5956c451] FINISH, GetHardwareInfoVDSCommand, log id: 670855a8 >2014-04-30 00:45:00,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [5956c451] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:45:00,893 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [319f09b1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:45:00,894 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [23732309] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:45:00,895 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) [23732309] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5732b74 >2014-04-30 00:45:00,908 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [23732309] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:45:00,909 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [23732309] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:45:02,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [2f6b8212] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 584f5886 >2014-04-30 00:45:03,009 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [2f6b8212] Failed in GetHardwareInfoVDS method >2014-04-30 00:45:03,009 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [2f6b8212] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:45:03,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [2f6b8212] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@413cf2b5 >2014-04-30 00:45:03,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [2f6b8212] HostName = 10.34.27.209 >2014-04-30 00:45:03,009 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [2f6b8212] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:45:03,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [2f6b8212] FINISH, GetHardwareInfoVDSCommand, log id: 584f5886 >2014-04-30 00:45:03,014 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [2f6b8212] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:45:03,025 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [2f6b8212] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:45:03,248 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-91) [6cf414a4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:45:03,250 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [15b88e77] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:45:03,256 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [15b88e77] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 14461f6f >2014-04-30 00:48:03,258 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [15b88e77] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:48:03,258 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [15b88e77] FINISH, ConnectStorageServerVDSCommand, log id: 14461f6f >2014-04-30 00:48:03,258 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [15b88e77] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:48:03,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 51ba9089 >2014-04-30 00:48:03,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 51ba9089 >2014-04-30 00:48:03,705 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:48:03,716 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-91) [7bb458d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:48:03,717 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) [7bb458d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2dd8e563 >2014-04-30 00:48:03,719 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) [7bb458d] FINISH, SetVdsStatusVDSCommand, log id: 2dd8e563 >2014-04-30 00:48:03,727 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [7bb458d] Correlation ID: 7bb458d, Job ID: 414a1f70-52a2-4bd1-ae72-4c9620c79a3b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:48:03,734 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [7bb458d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:48:03,737 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [7bb458d] Correlation ID: 6cf414a4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:48:03,741 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [49ac1fc6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:48:03,742 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [1c82b147] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:48:03,746 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) [1c82b147] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-37) [73412e9e] Autorecovering 1 hosts >2014-04-30 00:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-37) [73412e9e] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 73346336 >2014-04-30 00:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] FINISH, SetVdsStatusVDSCommand, log id: 73346336 >2014-04-30 00:50:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: c950279 >2014-04-30 00:50:00,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7bc9c588 >2014-04-30 00:50:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] Failed in GetHardwareInfoVDS method >2014-04-30 00:50:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:50:00,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7bc7a335 >2014-04-30 00:50:00,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] HostName = 10.34.27.209 >2014-04-30 00:50:00,869 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:50:00,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [360b80b1] FINISH, GetHardwareInfoVDSCommand, log id: 7bc9c588 >2014-04-30 00:50:00,878 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [360b80b1] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:50:00,889 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-37) [33a74ca3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:50:00,894 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:50:00,895 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: c950279 >2014-04-30 00:50:00,921 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:50:00,922 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:50:04,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5dad83cf] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 68cbafd1 >2014-04-30 00:50:04,339 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5dad83cf] Failed in GetHardwareInfoVDS method >2014-04-30 00:50:04,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5dad83cf] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:50:04,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5dad83cf] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1483f817 >2014-04-30 00:50:04,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5dad83cf] HostName = 10.34.27.209 >2014-04-30 00:50:04,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5dad83cf] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:50:04,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [5dad83cf] FINISH, GetHardwareInfoVDSCommand, log id: 68cbafd1 >2014-04-30 00:50:04,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [5dad83cf] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:50:04,356 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [5dad83cf] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:50:04,578 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-48) [4c77fbd0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:50:04,580 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [6192c953] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:50:04,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [6192c953] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: dab587c >2014-04-30 00:53:04,588 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [6192c953] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:53:04,588 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [6192c953] FINISH, ConnectStorageServerVDSCommand, log id: dab587c >2014-04-30 00:53:04,588 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [6192c953] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:53:04,592 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 23e7203 >2014-04-30 00:53:05,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 23e7203 >2014-04-30 00:53:05,033 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:53:05,044 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-48) [61cacbec] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:53:05,045 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [61cacbec] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 788cedb2 >2014-04-30 00:53:05,048 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [61cacbec] FINISH, SetVdsStatusVDSCommand, log id: 788cedb2 >2014-04-30 00:53:05,055 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [61cacbec] Correlation ID: 61cacbec, Job ID: f95f02a5-2787-43c4-97e7-a0e2463c0aed, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:53:05,061 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [61cacbec] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:53:05,064 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [61cacbec] Correlation ID: 4c77fbd0, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:53:05,068 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [3070ca13] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:53:05,069 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [7fa0aac6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:53:05,070 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [7fa0aac6] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 00:55:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [42ccfee2] Autorecovering 1 hosts >2014-04-30 00:55:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [42ccfee2] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 00:55:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 00:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 00:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 65e1eb88 >2014-04-30 00:55:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] FINISH, SetVdsStatusVDSCommand, log id: 65e1eb88 >2014-04-30 00:55:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 26fc8dc3 >2014-04-30 00:55:00,646 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 33277283 >2014-04-30 00:55:00,882 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] Failed in GetHardwareInfoVDS method >2014-04-30 00:55:00,882 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:55:00,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@62cf10b1 >2014-04-30 00:55:00,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] HostName = 10.34.27.209 >2014-04-30 00:55:00,882 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:55:00,882 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [266ccd2c] FINISH, GetHardwareInfoVDSCommand, log id: 33277283 >2014-04-30 00:55:00,888 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [266ccd2c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:55:00,898 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-56) [fa2e21e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:55:00,899 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-56) [65b46d31] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:55:00,900 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [65b46d31] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 26fc8dc3 >2014-04-30 00:55:00,914 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [65b46d31] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 00:55:00,915 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [65b46d31] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 00:55:02,441 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 63edb706 >2014-04-30 00:55:02,674 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] Failed in GetHardwareInfoVDS method >2014-04-30 00:55:02,674 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:55:02,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@b8b1e2b >2014-04-30 00:55:02,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] HostName = 10.34.27.209 >2014-04-30 00:55:02,674 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 00:55:02,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [2210c9fe] FINISH, GetHardwareInfoVDSCommand, log id: 63edb706 >2014-04-30 00:55:02,679 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [2210c9fe] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 00:55:02,689 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [2210c9fe] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 00:55:02,922 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-32) [5d749573] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:55:02,924 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-32) [26e868e2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 00:55:02,932 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [26e868e2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 38b10091 >2014-04-30 00:58:02,933 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [26e868e2] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 00:58:02,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [26e868e2] FINISH, ConnectStorageServerVDSCommand, log id: 38b10091 >2014-04-30 00:58:02,933 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-32) [26e868e2] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 00:58:02,937 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3ecfce98 >2014-04-30 00:58:03,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 3ecfce98 >2014-04-30 00:58:03,386 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 00:58:03,396 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-32) [2a0e065f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:58:03,397 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [2a0e065f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1e8eb204 >2014-04-30 00:58:03,400 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [2a0e065f] FINISH, SetVdsStatusVDSCommand, log id: 1e8eb204 >2014-04-30 00:58:03,408 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [2a0e065f] Correlation ID: 2a0e065f, Job ID: 85dea50f-0fbc-481c-8e2c-d53300391f9e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 00:58:03,415 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [2a0e065f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 00:58:03,418 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [2a0e065f] Correlation ID: 5d749573, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 00:58:03,424 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-32) [bf15d04] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:58:03,425 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-32) [7743b53d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 00:58:03,425 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [7743b53d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-43) [18044745] Autorecovering 1 hosts >2014-04-30 01:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-43) [18044745] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2d287cd0 >2014-04-30 01:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] FINISH, SetVdsStatusVDSCommand, log id: 2d287cd0 >2014-04-30 01:00:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6964bfe >2014-04-30 01:00:00,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 369bc456 >2014-04-30 01:00:00,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] Failed in GetHardwareInfoVDS method >2014-04-30 01:00:00,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:00:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2c77061c >2014-04-30 01:00:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] HostName = 10.34.27.209 >2014-04-30 01:00:00,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:00:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [72c583ea] FINISH, GetHardwareInfoVDSCommand, log id: 369bc456 >2014-04-30 01:00:00,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [72c583ea] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:00:00,918 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [3ea380d6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:00:00,919 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [20e5daa5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:00:00,920 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [20e5daa5] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6964bfe >2014-04-30 01:00:00,944 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [20e5daa5] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:00:00,945 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [20e5daa5] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:00:03,778 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1b1a0bab >2014-04-30 01:00:04,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] Failed in GetHardwareInfoVDS method >2014-04-30 01:00:04,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:00:04,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4af41573 >2014-04-30 01:00:04,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] HostName = 10.34.27.209 >2014-04-30 01:00:04,004 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:00:04,004 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [9c1882b] FINISH, GetHardwareInfoVDSCommand, log id: 1b1a0bab >2014-04-30 01:00:04,009 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [9c1882b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:00:04,018 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [9c1882b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:00:04,239 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-7) [4dbbdac] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:00:04,241 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [3933f405] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:00:04,249 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [3933f405] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 601c12da >2014-04-30 01:03:04,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [3933f405] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:03:04,251 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [3933f405] FINISH, ConnectStorageServerVDSCommand, log id: 601c12da >2014-04-30 01:03:04,251 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [3933f405] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:03:04,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 103a7ce8 >2014-04-30 01:03:04,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 103a7ce8 >2014-04-30 01:03:04,706 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:03:04,716 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-7) [4eecd2e8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:03:04,717 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [4eecd2e8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1b24e734 >2014-04-30 01:03:04,719 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [4eecd2e8] FINISH, SetVdsStatusVDSCommand, log id: 1b24e734 >2014-04-30 01:03:04,725 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [4eecd2e8] Correlation ID: 4eecd2e8, Job ID: a17efde2-7e66-4c88-8fa2-d030747df1c7, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:03:04,732 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [4eecd2e8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:03:04,734 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [4eecd2e8] Correlation ID: 4dbbdac, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:03:04,737 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-7) [425456ed] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:03:04,738 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [1be667d6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:03:04,739 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [1be667d6] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-36) [7476700d] Autorecovering 1 hosts >2014-04-30 01:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-36) [7476700d] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3b5afc67 >2014-04-30 01:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] FINISH, SetVdsStatusVDSCommand, log id: 3b5afc67 >2014-04-30 01:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 10c62c35 >2014-04-30 01:05:00,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2fb57047 >2014-04-30 01:05:00,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] Failed in GetHardwareInfoVDS method >2014-04-30 01:05:00,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:05:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@30de5dc9 >2014-04-30 01:05:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] HostName = 10.34.27.209 >2014-04-30 01:05:00,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:05:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [34fbacca] FINISH, GetHardwareInfoVDSCommand, log id: 2fb57047 >2014-04-30 01:05:00,871 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [34fbacca] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:05:00,879 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [189411e3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:05:00,880 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:05:00,881 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 10c62c35 >2014-04-30 01:05:00,894 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:05:00,895 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:05:02,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [3bf9c883] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 61e879f1 >2014-04-30 01:05:02,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [3bf9c883] Failed in GetHardwareInfoVDS method >2014-04-30 01:05:02,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [3bf9c883] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:05:02,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [3bf9c883] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@659def6e >2014-04-30 01:05:02,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [3bf9c883] HostName = 10.34.27.209 >2014-04-30 01:05:02,320 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [3bf9c883] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:05:02,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [3bf9c883] FINISH, GetHardwareInfoVDSCommand, log id: 61e879f1 >2014-04-30 01:05:02,325 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [3bf9c883] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:05:02,334 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [3bf9c883] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:05:02,556 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-19) [476e3858] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:05:02,558 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [15549ee9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:05:02,566 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [15549ee9] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6d2370f4 >2014-04-30 01:08:02,568 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [15549ee9] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:08:02,568 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [15549ee9] FINISH, ConnectStorageServerVDSCommand, log id: 6d2370f4 >2014-04-30 01:08:02,568 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [15549ee9] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:08:02,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 81ad67 >2014-04-30 01:08:03,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 81ad67 >2014-04-30 01:08:03,014 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:08:03,024 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-19) [1975c0e4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:08:03,025 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [1975c0e4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5bcd27bc >2014-04-30 01:08:03,028 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [1975c0e4] FINISH, SetVdsStatusVDSCommand, log id: 5bcd27bc >2014-04-30 01:08:03,035 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [1975c0e4] Correlation ID: 1975c0e4, Job ID: a77cb0b1-97ff-4bb0-9024-ff43d5e22936, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:08:03,042 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [1975c0e4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:08:03,045 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [1975c0e4] Correlation ID: 476e3858, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:08:03,050 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-19) [6cf900d6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:08:03,051 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-19) [522d21f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:08:03,051 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [522d21f3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-45) [3d2b8079] Autorecovering 1 hosts >2014-04-30 01:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-45) [3d2b8079] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 429ab98 >2014-04-30 01:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] FINISH, SetVdsStatusVDSCommand, log id: 429ab98 >2014-04-30 01:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7b0fc7 >2014-04-30 01:10:00,677 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1efb0a55 >2014-04-30 01:10:00,899 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] Failed in GetHardwareInfoVDS method >2014-04-30 01:10:00,899 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:10:00,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@73636256 >2014-04-30 01:10:00,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] HostName = 10.34.27.209 >2014-04-30 01:10:00,900 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:10:00,900 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [1eecae61] FINISH, GetHardwareInfoVDSCommand, log id: 1efb0a55 >2014-04-30 01:10:00,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [1eecae61] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:10:00,937 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [54c2230f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:10:00,939 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:10:00,940 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7b0fc7 >2014-04-30 01:10:00,955 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:10:00,956 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:10:03,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [1aa55f29] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 643bb7ea >2014-04-30 01:10:03,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [1aa55f29] Failed in GetHardwareInfoVDS method >2014-04-30 01:10:03,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [1aa55f29] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:10:03,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [1aa55f29] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@ca7970f >2014-04-30 01:10:03,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [1aa55f29] HostName = 10.34.27.209 >2014-04-30 01:10:03,620 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [1aa55f29] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:10:03,620 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [1aa55f29] FINISH, GetHardwareInfoVDSCommand, log id: 643bb7ea >2014-04-30 01:10:03,624 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [1aa55f29] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:10:03,633 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [1aa55f29] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:10:03,856 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-38) [4369ab0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:10:03,857 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-38) [7d82e295] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:10:03,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [7d82e295] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2bacfd62 >2014-04-30 01:13:03,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [7d82e295] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:13:03,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-38) [7d82e295] FINISH, ConnectStorageServerVDSCommand, log id: 2bacfd62 >2014-04-30 01:13:03,868 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-38) [7d82e295] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:13:03,880 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3ec1c47c >2014-04-30 01:13:04,320 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3ec1c47c >2014-04-30 01:13:04,320 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:13:04,330 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-38) [1243e63d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:13:04,331 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [1243e63d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 388e88c3 >2014-04-30 01:13:04,335 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [1243e63d] FINISH, SetVdsStatusVDSCommand, log id: 388e88c3 >2014-04-30 01:13:04,340 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [1243e63d] Correlation ID: 1243e63d, Job ID: 01ef375a-7bb4-4c43-ad8b-ed661683bcf1, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:13:04,347 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [1243e63d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:13:04,350 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [1243e63d] Correlation ID: 4369ab0, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:13:04,353 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-38) [58e5362a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:13:04,355 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-38) [4c9c7400] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:13:04,355 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-38) [4c9c7400] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [470b0a4] Autorecovering 1 hosts >2014-04-30 01:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-33) [470b0a4] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 53e61368 >2014-04-30 01:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] FINISH, SetVdsStatusVDSCommand, log id: 53e61368 >2014-04-30 01:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 698cdded >2014-04-30 01:15:00,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 627cf3c9 >2014-04-30 01:15:00,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] Failed in GetHardwareInfoVDS method >2014-04-30 01:15:00,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:15:00,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@796943a2 >2014-04-30 01:15:00,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] HostName = 10.34.27.209 >2014-04-30 01:15:00,868 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:15:00,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-33) [1c228d91] FINISH, GetHardwareInfoVDSCommand, log id: 627cf3c9 >2014-04-30 01:15:00,874 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-33) [1c228d91] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:15:00,882 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-33) [1cee6ca5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:15:00,883 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-33) [2c238432] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:15:00,884 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-33) [2c238432] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 698cdded >2014-04-30 01:15:00,899 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [2c238432] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:15:00,900 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-33) [2c238432] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:15:01,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [4974a741] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 514e7454 >2014-04-30 01:15:01,952 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [4974a741] Failed in GetHardwareInfoVDS method >2014-04-30 01:15:01,952 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [4974a741] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:15:01,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [4974a741] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@75b7607f >2014-04-30 01:15:01,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [4974a741] HostName = 10.34.27.209 >2014-04-30 01:15:01,952 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [4974a741] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:15:01,952 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [4974a741] FINISH, GetHardwareInfoVDSCommand, log id: 514e7454 >2014-04-30 01:15:01,957 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [4974a741] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:15:01,966 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [4974a741] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:15:02,188 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-59) [2387b369] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:15:02,189 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-59) [38bc85f4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:15:02,200 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [38bc85f4] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 74241260 >2014-04-30 01:18:02,201 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [38bc85f4] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:18:02,201 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [38bc85f4] FINISH, ConnectStorageServerVDSCommand, log id: 74241260 >2014-04-30 01:18:02,201 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-59) [38bc85f4] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:18:02,205 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3a3ab955 >2014-04-30 01:18:02,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3a3ab955 >2014-04-30 01:18:02,654 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:18:02,668 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-59) [34205c7f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:18:02,669 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [34205c7f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 67aa56ac >2014-04-30 01:18:02,671 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [34205c7f] FINISH, SetVdsStatusVDSCommand, log id: 67aa56ac >2014-04-30 01:18:02,681 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [34205c7f] Correlation ID: 34205c7f, Job ID: 39bd1ddd-3916-4348-9325-85cc2ab58f70, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:18:02,688 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [34205c7f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:18:02,690 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [34205c7f] Correlation ID: 2387b369, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:18:02,694 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-59) [8bfc634] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:18:02,695 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-59) [5b883a5a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:18:02,696 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) [5b883a5a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-73) [227e7ef] Autorecovering 1 hosts >2014-04-30 01:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-73) [227e7ef] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:20:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2af6b4af >2014-04-30 01:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] FINISH, SetVdsStatusVDSCommand, log id: 2af6b4af >2014-04-30 01:20:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 84b7e27 >2014-04-30 01:20:00,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 138001b0 >2014-04-30 01:20:00,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] Failed in GetHardwareInfoVDS method >2014-04-30 01:20:00,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:20:00,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@38eed3d0 >2014-04-30 01:20:00,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] HostName = 10.34.27.209 >2014-04-30 01:20:00,872 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:20:00,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [6aa608bc] FINISH, GetHardwareInfoVDSCommand, log id: 138001b0 >2014-04-30 01:20:00,877 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [6aa608bc] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:20:00,886 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [a250699] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:20:00,887 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:20:00,888 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 84b7e27 >2014-04-30 01:20:00,903 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:20:00,903 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:20:03,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [365f6e6d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 55c2c794 >2014-04-30 01:20:03,300 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [365f6e6d] Failed in GetHardwareInfoVDS method >2014-04-30 01:20:03,300 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [365f6e6d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:20:03,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [365f6e6d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@66bcf188 >2014-04-30 01:20:03,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [365f6e6d] HostName = 10.34.27.209 >2014-04-30 01:20:03,300 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [365f6e6d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:20:03,300 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [365f6e6d] FINISH, GetHardwareInfoVDSCommand, log id: 55c2c794 >2014-04-30 01:20:03,305 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [365f6e6d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:20:03,313 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [365f6e6d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:20:03,535 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-64) [301a928] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:20:03,536 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [33271f19] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:20:03,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [33271f19] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3a14a946 >2014-04-30 01:23:03,544 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [33271f19] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:23:03,544 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [33271f19] FINISH, ConnectStorageServerVDSCommand, log id: 3a14a946 >2014-04-30 01:23:03,544 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [33271f19] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:23:03,548 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 35003153 >2014-04-30 01:23:03,999 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 35003153 >2014-04-30 01:23:03,999 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:23:04,009 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-64) [7a560946] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:23:04,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [7a560946] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 58d23b3 >2014-04-30 01:23:04,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [7a560946] FINISH, SetVdsStatusVDSCommand, log id: 58d23b3 >2014-04-30 01:23:04,034 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [7a560946] Correlation ID: 7a560946, Job ID: dbac3741-e3a7-4678-a727-5cc8bbef3600, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:23:04,042 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [7a560946] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:23:04,043 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [7a560946] Correlation ID: 301a928, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:23:04,047 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [3ddbb0dc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:23:04,048 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [72405ec4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:23:04,049 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [72405ec4] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) [21d92d5f] Autorecovering 1 hosts >2014-04-30 01:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-87) [21d92d5f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5287b0aa >2014-04-30 01:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] FINISH, SetVdsStatusVDSCommand, log id: 5287b0aa >2014-04-30 01:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 719eb200 >2014-04-30 01:25:00,645 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 46ce9b81 >2014-04-30 01:25:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] Failed in GetHardwareInfoVDS method >2014-04-30 01:25:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:25:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@e5928f2 >2014-04-30 01:25:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] HostName = 10.34.27.209 >2014-04-30 01:25:00,879 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:25:00,879 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-87) [1914c913] FINISH, GetHardwareInfoVDSCommand, log id: 46ce9b81 >2014-04-30 01:25:00,887 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-87) [1914c913] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:25:00,898 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-87) [1302d718] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:25:00,900 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-87) [4d3c9739] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:25:00,901 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-87) [4d3c9739] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 719eb200 >2014-04-30 01:25:00,924 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [4d3c9739] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:25:00,926 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-87) [4d3c9739] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:25:01,400 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7e680b09] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1589c219 >2014-04-30 01:25:01,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7e680b09] Failed in GetHardwareInfoVDS method >2014-04-30 01:25:01,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7e680b09] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:25:01,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7e680b09] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@726025b >2014-04-30 01:25:01,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7e680b09] HostName = 10.34.27.209 >2014-04-30 01:25:01,622 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7e680b09] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:25:01,622 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [7e680b09] FINISH, GetHardwareInfoVDSCommand, log id: 1589c219 >2014-04-30 01:25:01,627 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [7e680b09] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:25:01,636 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [7e680b09] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:25:01,859 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-69) [5fbc4930] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:25:01,860 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [49e75c1f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:25:01,866 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [49e75c1f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 9f6fbe2 >2014-04-30 01:28:01,867 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [49e75c1f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:28:01,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [49e75c1f] FINISH, ConnectStorageServerVDSCommand, log id: 9f6fbe2 >2014-04-30 01:28:01,868 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [49e75c1f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:28:01,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 2b7e2c73 >2014-04-30 01:28:02,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 2b7e2c73 >2014-04-30 01:28:02,322 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:28:02,332 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-69) [1c0a5f8f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:28:02,334 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [1c0a5f8f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: d99a796 >2014-04-30 01:28:02,336 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [1c0a5f8f] FINISH, SetVdsStatusVDSCommand, log id: d99a796 >2014-04-30 01:28:02,342 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [1c0a5f8f] Correlation ID: 1c0a5f8f, Job ID: 975d18a7-ce2b-4286-b962-46b263c944d4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:28:02,350 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [1c0a5f8f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:28:02,353 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [1c0a5f8f] Correlation ID: 5fbc4930, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:28:02,358 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-69) [46a732d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:28:02,359 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-69) [59b47c84] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:28:02,359 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [59b47c84] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-4) [56d5ef3f] Autorecovering 1 hosts >2014-04-30 01:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-4) [56d5ef3f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:30:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 216674d9 >2014-04-30 01:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] FINISH, SetVdsStatusVDSCommand, log id: 216674d9 >2014-04-30 01:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7ee0964b >2014-04-30 01:30:00,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 24b023ec >2014-04-30 01:30:00,950 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] Failed in GetHardwareInfoVDS method >2014-04-30 01:30:00,951 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:30:00,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@12f18100 >2014-04-30 01:30:00,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] HostName = 10.34.27.209 >2014-04-30 01:30:00,951 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:30:00,951 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4de50e8] FINISH, GetHardwareInfoVDSCommand, log id: 24b023ec >2014-04-30 01:30:00,957 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [4de50e8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:30:00,966 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-4) [2962acc9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:30:00,968 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-4) [316204bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:30:00,968 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [316204bc] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7ee0964b >2014-04-30 01:30:00,990 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [316204bc] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:30:00,991 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [316204bc] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:30:02,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6a119ac2 >2014-04-30 01:30:02,938 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] Failed in GetHardwareInfoVDS method >2014-04-30 01:30:02,938 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:30:02,938 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5235d189 >2014-04-30 01:30:02,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] HostName = 10.34.27.209 >2014-04-30 01:30:02,939 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:30:02,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [37aba6ef] FINISH, GetHardwareInfoVDSCommand, log id: 6a119ac2 >2014-04-30 01:30:02,943 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [37aba6ef] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:30:02,952 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [37aba6ef] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:30:03,174 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-3) [2f170541] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:30:03,176 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-3) [22d29c8a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:30:03,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-3) [22d29c8a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 52bfaa0e >2014-04-30 01:33:03,184 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-3) [22d29c8a] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:33:03,184 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-3) [22d29c8a] FINISH, ConnectStorageServerVDSCommand, log id: 52bfaa0e >2014-04-30 01:33:03,184 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-3) [22d29c8a] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:33:03,187 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 11e672fb >2014-04-30 01:33:03,664 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 11e672fb >2014-04-30 01:33:03,664 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:33:03,676 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-3) [6edf5ed9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:33:03,677 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [6edf5ed9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 211fcafd >2014-04-30 01:33:03,679 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [6edf5ed9] FINISH, SetVdsStatusVDSCommand, log id: 211fcafd >2014-04-30 01:33:03,685 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [6edf5ed9] Correlation ID: 6edf5ed9, Job ID: b3a3f690-4628-4933-8100-f89adadbb5b0, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:33:03,692 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [6edf5ed9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:33:03,695 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [6edf5ed9] Correlation ID: 2f170541, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:33:03,699 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-3) [aa26a98] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:33:03,700 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:33:03,701 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-11) [4f4ccdd1] Autorecovering 1 hosts >2014-04-30 01:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-11) [4f4ccdd1] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4609ef19 >2014-04-30 01:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] FINISH, SetVdsStatusVDSCommand, log id: 4609ef19 >2014-04-30 01:35:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 453449e3 >2014-04-30 01:35:00,643 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 9f6e28e >2014-04-30 01:35:00,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] Failed in GetHardwareInfoVDS method >2014-04-30 01:35:00,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:35:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2f93ddbe >2014-04-30 01:35:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] HostName = 10.34.27.209 >2014-04-30 01:35:00,865 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:35:00,865 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [6a55867a] FINISH, GetHardwareInfoVDSCommand, log id: 9f6e28e >2014-04-30 01:35:00,870 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [6a55867a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:35:00,878 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-11) [7f5e1642] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:35:00,879 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-11) [398f8810] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:35:00,880 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) [398f8810] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 453449e3 >2014-04-30 01:35:00,894 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [398f8810] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:35:00,895 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [398f8810] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:35:04,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [23732309] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: a07c817 >2014-04-30 01:35:04,278 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [23732309] Failed in GetHardwareInfoVDS method >2014-04-30 01:35:04,278 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [23732309] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:35:04,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [23732309] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4e16326 >2014-04-30 01:35:04,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [23732309] HostName = 10.34.27.209 >2014-04-30 01:35:04,279 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [23732309] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:35:04,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [23732309] FINISH, GetHardwareInfoVDSCommand, log id: a07c817 >2014-04-30 01:35:04,283 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [23732309] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:35:04,292 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [23732309] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:35:04,513 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-93) [c65e010] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:35:04,515 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [39233dd1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:35:04,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [39233dd1] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 32dcd668 >2014-04-30 01:36:05,015 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-89) [7b33c831] Start refreshing all users data >2014-04-30 01:36:07,293 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [4badcb49] Attempting to update VMs/Templates Ovf. >2014-04-30 01:36:07,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [4badcb49] Attempting to update VM OVFs in Data Center NFS >2014-04-30 01:36:07,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [4badcb49] Successfully updated VM OVFs in Data Center NFS >2014-04-30 01:36:07,294 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [4badcb49] Attempting to update template OVFs in Data Center NFS >2014-04-30 01:36:07,295 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [4badcb49] Successfully updated templates OVFs in Data Center NFS >2014-04-30 01:36:07,295 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [4badcb49] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-30 01:36:07,296 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-60) [4badcb49] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-30 01:38:04,523 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [39233dd1] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:38:04,523 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [39233dd1] FINISH, ConnectStorageServerVDSCommand, log id: 32dcd668 >2014-04-30 01:38:04,523 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [39233dd1] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:38:04,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 465eb106 >2014-04-30 01:38:04,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 465eb106 >2014-04-30 01:38:04,978 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:38:04,989 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-93) [2159c3c9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:38:04,990 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [2159c3c9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1475f0a2 >2014-04-30 01:38:04,992 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [2159c3c9] FINISH, SetVdsStatusVDSCommand, log id: 1475f0a2 >2014-04-30 01:38:04,999 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [2159c3c9] Correlation ID: 2159c3c9, Job ID: 09e63b6d-46fd-4825-b6e5-2c2a674e3fba, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:38:05,005 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [2159c3c9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:38:05,007 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [2159c3c9] Correlation ID: c65e010, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:38:05,011 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [1ffa2e51] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:38:05,013 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [2126fd16] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:38:05,013 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [2126fd16] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-21) [413d6449] Autorecovering 1 hosts >2014-04-30 01:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-21) [413d6449] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:40:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:40:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5648397a >2014-04-30 01:40:00,019 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] FINISH, SetVdsStatusVDSCommand, log id: 5648397a >2014-04-30 01:40:00,042 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6d6ce691 >2014-04-30 01:40:00,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6259c6a6 >2014-04-30 01:40:00,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] Failed in GetHardwareInfoVDS method >2014-04-30 01:40:00,897 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:40:00,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@99fd374 >2014-04-30 01:40:00,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] HostName = 10.34.27.209 >2014-04-30 01:40:00,898 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:40:00,898 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [265fec99] FINISH, GetHardwareInfoVDSCommand, log id: 6259c6a6 >2014-04-30 01:40:00,902 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [265fec99] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:40:00,909 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-21) [3fc38c45] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:40:00,910 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-21) [cb00b75] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:40:00,911 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) [cb00b75] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6d6ce691 >2014-04-30 01:40:00,924 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [cb00b75] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:40:00,925 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [cb00b75] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:40:02,364 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 10ee4f67 >2014-04-30 01:40:02,587 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] Failed in GetHardwareInfoVDS method >2014-04-30 01:40:02,587 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:40:02,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@d9efbe0 >2014-04-30 01:40:02,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] HostName = 10.34.27.209 >2014-04-30 01:40:02,587 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:40:02,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [7060185e] FINISH, GetHardwareInfoVDSCommand, log id: 10ee4f67 >2014-04-30 01:40:02,592 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [7060185e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:40:02,600 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [7060185e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:40:02,822 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-17) [76a38fab] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:40:02,824 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-17) [299de3cd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:40:02,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-17) [299de3cd] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5d46d623 >2014-04-30 01:43:02,833 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-17) [299de3cd] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:43:02,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-17) [299de3cd] FINISH, ConnectStorageServerVDSCommand, log id: 5d46d623 >2014-04-30 01:43:02,833 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-17) [299de3cd] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:43:02,839 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7616047c >2014-04-30 01:43:03,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 7616047c >2014-04-30 01:43:03,279 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:43:03,289 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-17) [61a78cbb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:43:03,290 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [61a78cbb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2afb014d >2014-04-30 01:43:03,292 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [61a78cbb] FINISH, SetVdsStatusVDSCommand, log id: 2afb014d >2014-04-30 01:43:03,298 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [61a78cbb] Correlation ID: 61a78cbb, Job ID: 3c110086-3c9a-4213-897a-324a72a9d6c1, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:43:03,304 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [61a78cbb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:43:03,306 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [61a78cbb] Correlation ID: 76a38fab, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:43:03,310 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-17) [227d2b4b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:43:03,310 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-17) [30691e32] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:43:03,311 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-17) [30691e32] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-42) [796f9661] Autorecovering 1 hosts >2014-04-30 01:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-42) [796f9661] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 49aede7a >2014-04-30 01:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] FINISH, SetVdsStatusVDSCommand, log id: 49aede7a >2014-04-30 01:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6619d00e >2014-04-30 01:45:00,654 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2937c01d >2014-04-30 01:45:00,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] Failed in GetHardwareInfoVDS method >2014-04-30 01:45:00,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:45:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@b25c50f >2014-04-30 01:45:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] HostName = 10.34.27.209 >2014-04-30 01:45:00,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:45:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [442abca8] FINISH, GetHardwareInfoVDSCommand, log id: 2937c01d >2014-04-30 01:45:00,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [442abca8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:45:00,891 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-42) [36833eaf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:45:00,892 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-42) [29340fe2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:45:00,893 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) [29340fe2] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6619d00e >2014-04-30 01:45:00,906 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [29340fe2] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:45:00,907 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [29340fe2] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:45:03,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [7e9eb086] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5494fbf4 >2014-04-30 01:45:03,891 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [7e9eb086] Failed in GetHardwareInfoVDS method >2014-04-30 01:45:03,892 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [7e9eb086] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:45:03,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [7e9eb086] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1297b8e >2014-04-30 01:45:03,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [7e9eb086] HostName = 10.34.27.209 >2014-04-30 01:45:03,892 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [7e9eb086] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:45:03,892 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [7e9eb086] FINISH, GetHardwareInfoVDSCommand, log id: 5494fbf4 >2014-04-30 01:45:03,897 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [7e9eb086] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:45:03,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [7e9eb086] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:45:04,126 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-92) [6c862530] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:45:04,128 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-92) [58db0097] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:45:04,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [58db0097] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: aaaff8d >2014-04-30 01:48:04,135 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [58db0097] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:48:04,135 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-92) [58db0097] FINISH, ConnectStorageServerVDSCommand, log id: aaaff8d >2014-04-30 01:48:04,135 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-92) [58db0097] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:48:04,139 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 32afb57e >2014-04-30 01:48:04,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 32afb57e >2014-04-30 01:48:04,582 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:48:04,594 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-92) [2aafe31] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:48:04,596 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [2aafe31] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 39d62933 >2014-04-30 01:48:04,598 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [2aafe31] FINISH, SetVdsStatusVDSCommand, log id: 39d62933 >2014-04-30 01:48:04,605 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [2aafe31] Correlation ID: 2aafe31, Job ID: cdbf39d0-1641-4e4c-b0a0-89cf22a82a0b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:48:04,615 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [2aafe31] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:48:04,617 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [2aafe31] Correlation ID: 6c862530, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:48:04,623 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-92) [71796c2e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:48:04,624 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-92) [6b8f6d0c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:48:04,625 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-92) [6b8f6d0c] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [2b8adcc] Autorecovering 1 hosts >2014-04-30 01:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [2b8adcc] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 39646625 >2014-04-30 01:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] FINISH, SetVdsStatusVDSCommand, log id: 39646625 >2014-04-30 01:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2d242ea8 >2014-04-30 01:50:00,644 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 28ca6c78 >2014-04-30 01:50:00,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] Failed in GetHardwareInfoVDS method >2014-04-30 01:50:00,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:50:00,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1974f34 >2014-04-30 01:50:00,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] HostName = 10.34.27.209 >2014-04-30 01:50:00,876 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:50:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [2024168f] FINISH, GetHardwareInfoVDSCommand, log id: 28ca6c78 >2014-04-30 01:50:00,882 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [2024168f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:50:00,893 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [4a50c1bf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:50:00,895 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [5e757262] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:50:00,896 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [5e757262] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2d242ea8 >2014-04-30 01:50:00,915 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [5e757262] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:50:00,916 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [5e757262] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:50:01,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1be667d6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 283c8060 >2014-04-30 01:50:02,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1be667d6] Failed in GetHardwareInfoVDS method >2014-04-30 01:50:02,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1be667d6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:50:02,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1be667d6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1c103343 >2014-04-30 01:50:02,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1be667d6] HostName = 10.34.27.209 >2014-04-30 01:50:02,207 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1be667d6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:50:02,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [1be667d6] FINISH, GetHardwareInfoVDSCommand, log id: 283c8060 >2014-04-30 01:50:02,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [1be667d6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:50:02,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [1be667d6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:50:02,442 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-7) [71b4dba5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:50:02,444 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [70014378] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:50:02,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [70014378] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1effdd34 >2014-04-30 01:53:02,453 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [70014378] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:53:02,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [70014378] FINISH, ConnectStorageServerVDSCommand, log id: 1effdd34 >2014-04-30 01:53:02,453 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [70014378] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:53:02,457 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 50b94e2c >2014-04-30 01:53:02,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 50b94e2c >2014-04-30 01:53:02,899 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:53:02,908 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-7) [50330858] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:53:02,909 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [50330858] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 68c0933b >2014-04-30 01:53:02,912 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [50330858] FINISH, SetVdsStatusVDSCommand, log id: 68c0933b >2014-04-30 01:53:02,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [50330858] Correlation ID: 50330858, Job ID: 1b480af1-8944-472d-98e1-24a3e0e2dc2f, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:53:02,924 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [50330858] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:53:02,926 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [50330858] Correlation ID: 71b4dba5, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:53:02,929 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-7) [41aa3073] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:53:02,930 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:53:02,931 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 01:55:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [4b3610c4] Autorecovering 1 hosts >2014-04-30 01:55:00,007 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [4b3610c4] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 01:55:00,011 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:55:00,011 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:55:00,011 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 01:55:00,011 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 01:55:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 9127807 >2014-04-30 01:55:00,019 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] FINISH, SetVdsStatusVDSCommand, log id: 9127807 >2014-04-30 01:55:00,021 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 326501a9 >2014-04-30 01:55:00,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 75cee46a >2014-04-30 01:55:00,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] Failed in GetHardwareInfoVDS method >2014-04-30 01:55:00,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:55:00,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@61f62792 >2014-04-30 01:55:00,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] HostName = 10.34.27.209 >2014-04-30 01:55:00,875 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:55:00,876 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [72e933cb] FINISH, GetHardwareInfoVDSCommand, log id: 75cee46a >2014-04-30 01:55:00,881 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [72e933cb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:55:00,894 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [20ad31] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:55:00,896 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [1b2eb2bc] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:55:00,896 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [1b2eb2bc] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 326501a9 >2014-04-30 01:55:00,915 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [1b2eb2bc] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 01:55:00,916 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [1b2eb2bc] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 01:55:03,301 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [108d9d32] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 77c7711 >2014-04-30 01:55:03,524 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [108d9d32] Failed in GetHardwareInfoVDS method >2014-04-30 01:55:03,524 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [108d9d32] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:55:03,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [108d9d32] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7c6d00ed >2014-04-30 01:55:03,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [108d9d32] HostName = 10.34.27.209 >2014-04-30 01:55:03,524 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [108d9d32] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 01:55:03,524 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [108d9d32] FINISH, GetHardwareInfoVDSCommand, log id: 77c7711 >2014-04-30 01:55:03,528 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [108d9d32] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 01:55:03,537 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [108d9d32] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 01:55:03,759 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-82) [1c161031] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:55:03,760 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-82) [67e0184e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 01:55:03,769 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [67e0184e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 73d7f8ab >2014-04-30 01:58:03,771 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [67e0184e] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 01:58:03,771 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-82) [67e0184e] FINISH, ConnectStorageServerVDSCommand, log id: 73d7f8ab >2014-04-30 01:58:03,771 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-82) [67e0184e] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 01:58:03,775 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 52c1f284 >2014-04-30 01:58:04,216 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 52c1f284 >2014-04-30 01:58:04,216 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 01:58:04,229 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-82) [3078e9aa] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:58:04,231 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3078e9aa] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4bf1027c >2014-04-30 01:58:04,234 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [3078e9aa] FINISH, SetVdsStatusVDSCommand, log id: 4bf1027c >2014-04-30 01:58:04,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3078e9aa] Correlation ID: 3078e9aa, Job ID: 9d45abb0-441a-4a9d-ad5e-c50df40eb103, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 01:58:04,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3078e9aa] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 01:58:04,249 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [3078e9aa] Correlation ID: 1c161031, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 01:58:04,254 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-82) [62d5f07d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:58:04,255 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-82) [46150d65] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 01:58:04,256 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-82) [46150d65] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-95) [3376995f] Autorecovering 1 hosts >2014-04-30 02:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-95) [3376995f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:00:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:00:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:00:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:00:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 44bdd79e >2014-04-30 02:00:00,025 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] FINISH, SetVdsStatusVDSCommand, log id: 44bdd79e >2014-04-30 02:00:00,026 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 77f0e686 >2014-04-30 02:00:00,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 214bb072 >2014-04-30 02:00:00,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] Failed in GetHardwareInfoVDS method >2014-04-30 02:00:00,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:00:00,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7b08385e >2014-04-30 02:00:00,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] HostName = 10.34.27.209 >2014-04-30 02:00:00,881 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:00:00,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [59a19bd3] FINISH, GetHardwareInfoVDSCommand, log id: 214bb072 >2014-04-30 02:00:00,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [59a19bd3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:00:00,894 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [4b0d5f68] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:00:00,895 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [179ebe56] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:00:00,896 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) [179ebe56] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 77f0e686 >2014-04-30 02:00:00,909 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [179ebe56] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:00:00,911 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [179ebe56] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:00:01,607 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2e861d1b >2014-04-30 02:00:01,830 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] Failed in GetHardwareInfoVDS method >2014-04-30 02:00:01,830 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:00:01,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@66e0ef79 >2014-04-30 02:00:01,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] HostName = 10.34.27.209 >2014-04-30 02:00:01,831 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:00:01,831 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-97) [39f1717f] FINISH, GetHardwareInfoVDSCommand, log id: 2e861d1b >2014-04-30 02:00:01,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [39f1717f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:00:01,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [39f1717f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:00:02,069 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-97) [123a62a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:00:02,070 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-97) [771f37e3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:00:02,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-97) [771f37e3] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1307053d >2014-04-30 02:03:02,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-97) [771f37e3] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:03:02,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-97) [771f37e3] FINISH, ConnectStorageServerVDSCommand, log id: 1307053d >2014-04-30 02:03:02,080 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-97) [771f37e3] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:03:02,098 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7217fc55 >2014-04-30 02:03:02,558 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 7217fc55 >2014-04-30 02:03:02,558 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:03:02,568 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-97) [3630a10b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:03:02,569 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [3630a10b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6e8af172 >2014-04-30 02:03:02,571 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-97) [3630a10b] FINISH, SetVdsStatusVDSCommand, log id: 6e8af172 >2014-04-30 02:03:02,577 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [3630a10b] Correlation ID: 3630a10b, Job ID: 8f1529bd-e2ef-4f2e-a332-4e52165aaf7e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:03:02,584 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [3630a10b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:03:02,586 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-97) [3630a10b] Correlation ID: 123a62a, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:03:02,592 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-97) [f543097] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:03:02,593 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-97) [de1e14c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:03:02,594 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-97) [de1e14c] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-55) [531e7ffa] Autorecovering 1 hosts >2014-04-30 02:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-55) [531e7ffa] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:05:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5836a234 >2014-04-30 02:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] FINISH, SetVdsStatusVDSCommand, log id: 5836a234 >2014-04-30 02:05:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4abb6d67 >2014-04-30 02:05:00,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 62c10e57 >2014-04-30 02:05:00,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] Failed in GetHardwareInfoVDS method >2014-04-30 02:05:00,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:05:00,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7094a467 >2014-04-30 02:05:00,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] HostName = 10.34.27.209 >2014-04-30 02:05:00,871 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:05:00,871 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [490b322d] FINISH, GetHardwareInfoVDSCommand, log id: 62c10e57 >2014-04-30 02:05:00,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [490b322d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:05:00,884 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [1b92ea00] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:05:00,885 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:05:00,886 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4abb6d67 >2014-04-30 02:05:00,899 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:05:00,900 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:05:02,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4e51bada >2014-04-30 02:05:03,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] Failed in GetHardwareInfoVDS method >2014-04-30 02:05:03,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:05:03,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4dac9bca >2014-04-30 02:05:03,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] HostName = 10.34.27.209 >2014-04-30 02:05:03,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:05:03,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [1114b484] FINISH, GetHardwareInfoVDSCommand, log id: 4e51bada >2014-04-30 02:05:03,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [1114b484] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:05:03,200 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [1114b484] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:05:03,421 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-63) [13ff6d39] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:05:03,423 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-63) [7279d82] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:05:03,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-63) [7279d82] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 21586788 >2014-04-30 02:08:03,431 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-63) [7279d82] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:08:03,431 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-63) [7279d82] FINISH, ConnectStorageServerVDSCommand, log id: 21586788 >2014-04-30 02:08:03,431 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-63) [7279d82] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:08:03,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 37a599ba >2014-04-30 02:08:03,894 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 37a599ba >2014-04-30 02:08:03,894 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:08:03,906 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-63) [13ea9f0a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:08:03,907 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [13ea9f0a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 365e470 >2014-04-30 02:08:03,910 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [13ea9f0a] FINISH, SetVdsStatusVDSCommand, log id: 365e470 >2014-04-30 02:08:03,916 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [13ea9f0a] Correlation ID: 13ea9f0a, Job ID: d8c2bfd6-46a4-433e-882a-7fff1c712714, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:08:03,922 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [13ea9f0a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:08:03,924 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [13ea9f0a] Correlation ID: 13ff6d39, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:08:03,928 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-63) [519b1fac] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:08:03,929 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-63) [2af8f9ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:08:03,930 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) [2af8f9ae] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-98) [21fb7024] Autorecovering 1 hosts >2014-04-30 02:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-98) [21fb7024] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 745f2bcc >2014-04-30 02:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] FINISH, SetVdsStatusVDSCommand, log id: 745f2bcc >2014-04-30 02:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4cc4f4ed >2014-04-30 02:10:00,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2f9cbfda >2014-04-30 02:10:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] Failed in GetHardwareInfoVDS method >2014-04-30 02:10:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:10:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3a998759 >2014-04-30 02:10:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] HostName = 10.34.27.209 >2014-04-30 02:10:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:10:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [2af55f5f] FINISH, GetHardwareInfoVDSCommand, log id: 2f9cbfda >2014-04-30 02:10:00,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [2af55f5f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:10:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-98) [475af000] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:10:00,725 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-98) [48e5821f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:10:00,726 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) [48e5821f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4cc4f4ed >2014-04-30 02:10:00,746 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [48e5821f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:10:00,747 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [48e5821f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:10:01,279 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [72405ec4] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 58de6980 >2014-04-30 02:10:01,503 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [72405ec4] Failed in GetHardwareInfoVDS method >2014-04-30 02:10:01,503 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [72405ec4] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:10:01,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [72405ec4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@71e08014 >2014-04-30 02:10:01,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [72405ec4] HostName = 10.34.27.209 >2014-04-30 02:10:01,503 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [72405ec4] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:10:01,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [72405ec4] FINISH, GetHardwareInfoVDSCommand, log id: 58de6980 >2014-04-30 02:10:01,507 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [72405ec4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:10:01,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [72405ec4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:10:01,738 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-64) [6c685ad5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:10:01,739 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [23b16308] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:10:01,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [23b16308] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4c638ff3 >2014-04-30 02:13:01,747 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [23b16308] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:13:01,747 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [23b16308] FINISH, ConnectStorageServerVDSCommand, log id: 4c638ff3 >2014-04-30 02:13:01,748 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [23b16308] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:13:01,751 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 65f4c2fa >2014-04-30 02:13:02,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 65f4c2fa >2014-04-30 02:13:02,189 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:13:02,199 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-64) [4e574be4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:13:02,200 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [4e574be4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 362ace7c >2014-04-30 02:13:02,204 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [4e574be4] FINISH, SetVdsStatusVDSCommand, log id: 362ace7c >2014-04-30 02:13:02,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [4e574be4] Correlation ID: 4e574be4, Job ID: c8f3d460-67f9-4dd3-b1e6-8db0e5148150, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:13:02,216 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [4e574be4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:13:02,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [4e574be4] Correlation ID: 6c685ad5, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:13:02,221 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [6f20de24] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:13:02,222 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [303a359a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:13:02,223 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [303a359a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:15:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-35) [3b027fc1] Autorecovering 1 hosts >2014-04-30 02:15:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-35) [3b027fc1] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:15:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:15:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:15:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 401ca0ac >2014-04-30 02:15:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] FINISH, SetVdsStatusVDSCommand, log id: 401ca0ac >2014-04-30 02:15:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 593bc7f1 >2014-04-30 02:15:00,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6c247249 >2014-04-30 02:15:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] Failed in GetHardwareInfoVDS method >2014-04-30 02:15:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:15:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4ccb4e16 >2014-04-30 02:15:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] HostName = 10.34.27.209 >2014-04-30 02:15:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:15:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [1b602f30] FINISH, GetHardwareInfoVDSCommand, log id: 6c247249 >2014-04-30 02:15:00,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [1b602f30] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:15:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-35) [36e2348d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:15:00,723 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:15:00,724 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 593bc7f1 >2014-04-30 02:15:00,738 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:15:00,739 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:15:02,581 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 15b0be24 >2014-04-30 02:15:02,803 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) Failed in GetHardwareInfoVDS method >2014-04-30 02:15:02,803 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:15:02,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@20a0a01c >2014-04-30 02:15:02,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) HostName = 10.34.27.209 >2014-04-30 02:15:02,803 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:15:02,803 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) FINISH, GetHardwareInfoVDSCommand, log id: 15b0be24 >2014-04-30 02:15:02,808 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:15:02,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:15:03,042 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-70) [4bbed7ca] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:15:03,044 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-70) [3b3a5a89] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:15:03,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [3b3a5a89] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 20178422 >2014-04-30 02:18:03,052 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [3b3a5a89] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:18:03,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [3b3a5a89] FINISH, ConnectStorageServerVDSCommand, log id: 20178422 >2014-04-30 02:18:03,052 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-70) [3b3a5a89] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:18:03,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: d50dfdc >2014-04-30 02:18:03,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: d50dfdc >2014-04-30 02:18:03,496 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:18:03,508 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-70) [4af2da18] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:18:03,510 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [4af2da18] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: e8dad84 >2014-04-30 02:18:03,513 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [4af2da18] FINISH, SetVdsStatusVDSCommand, log id: e8dad84 >2014-04-30 02:18:03,519 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [4af2da18] Correlation ID: 4af2da18, Job ID: 2aad1b9a-f1c8-4bb0-a776-adda8c8fa395, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:18:03,530 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [4af2da18] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:18:03,533 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [4af2da18] Correlation ID: 4bbed7ca, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:18:03,540 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-70) [7c057066] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:18:03,541 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-70) [67291b24] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:18:03,543 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [67291b24] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) [46031e23] Autorecovering 1 hosts >2014-04-30 02:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-85) [46031e23] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [26241210] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [26241210] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [26241210] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [26241210] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [26241210] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: ee67a22 >2014-04-30 02:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [26241210] FINISH, SetVdsStatusVDSCommand, log id: ee67a22 >2014-04-30 02:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) [26241210] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2ef0a192 >2014-04-30 02:20:00,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [26241210] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3dddf99b >2014-04-30 02:20:00,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [26241210] Failed in GetHardwareInfoVDS method >2014-04-30 02:20:00,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [26241210] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:20:00,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [26241210] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@382e4ed8 >2014-04-30 02:20:00,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [26241210] HostName = 10.34.27.209 >2014-04-30 02:20:00,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [26241210] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:20:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [26241210] FINISH, GetHardwareInfoVDSCommand, log id: 3dddf99b >2014-04-30 02:20:00,721 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [26241210] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:20:00,734 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-85) [3e3ca538] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:20:00,735 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:20:00,736 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2ef0a192 >2014-04-30 02:20:00,754 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:20:00,756 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:20:03,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1ff1ceed >2014-04-30 02:20:04,121 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] Failed in GetHardwareInfoVDS method >2014-04-30 02:20:04,121 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:20:04,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@34185da5 >2014-04-30 02:20:04,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] HostName = 10.34.27.209 >2014-04-30 02:20:04,121 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:20:04,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] FINISH, GetHardwareInfoVDSCommand, log id: 1ff1ceed >2014-04-30 02:20:04,126 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:20:04,136 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [2dfd3fea] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:20:04,359 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-3) [6a33ddec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:20:04,361 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-3) [43773212] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:20:04,367 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-3) [43773212] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 58399af3 >2014-04-30 02:23:04,370 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-3) [43773212] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:23:04,370 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-3) [43773212] FINISH, ConnectStorageServerVDSCommand, log id: 58399af3 >2014-04-30 02:23:04,371 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-3) [43773212] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:23:04,375 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 2c93cbf1 >2014-04-30 02:23:04,819 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 2c93cbf1 >2014-04-30 02:23:04,819 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:23:04,830 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-3) [1da5c72e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:23:04,831 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [1da5c72e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3648cfff >2014-04-30 02:23:04,834 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-3) [1da5c72e] FINISH, SetVdsStatusVDSCommand, log id: 3648cfff >2014-04-30 02:23:04,839 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [1da5c72e] Correlation ID: 1da5c72e, Job ID: a24d67d2-fe71-4389-96f1-34150c1ef2f4, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:23:04,845 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [1da5c72e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:23:04,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-3) [1da5c72e] Correlation ID: 6a33ddec, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:23:04,851 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-3) [2da3447] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:23:04,852 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-3) [3fc1d029] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:23:04,853 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-3) [3fc1d029] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-61) [19c7c217] Autorecovering 1 hosts >2014-04-30 02:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-61) [19c7c217] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:25:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5c903c59 >2014-04-30 02:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] FINISH, SetVdsStatusVDSCommand, log id: 5c903c59 >2014-04-30 02:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 76dc3ec9 >2014-04-30 02:25:00,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7d0d586b >2014-04-30 02:25:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] Failed in GetHardwareInfoVDS method >2014-04-30 02:25:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:25:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@772818f0 >2014-04-30 02:25:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] HostName = 10.34.27.209 >2014-04-30 02:25:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:25:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [33b17abb] FINISH, GetHardwareInfoVDSCommand, log id: 7d0d586b >2014-04-30 02:25:00,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [33b17abb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:25:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-61) [6eb0ecce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:25:00,718 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:25:00,719 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 76dc3ec9 >2014-04-30 02:25:00,735 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:25:00,736 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:25:02,238 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b3fd331] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7126e41b >2014-04-30 02:25:02,461 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Failed in GetHardwareInfoVDS method >2014-04-30 02:25:02,461 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:25:02,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3049e2a6 >2014-04-30 02:25:02,461 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b3fd331] HostName = 10.34.27.209 >2014-04-30 02:25:02,462 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:25:02,462 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-30) [5b3fd331] FINISH, GetHardwareInfoVDSCommand, log id: 7126e41b >2014-04-30 02:25:02,466 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:25:02,474 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [5b3fd331] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:25:02,695 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-30) [6e4be29f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:25:02,697 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [5406c0d4] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:25:02,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [5406c0d4] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 43654283 >2014-04-30 02:28:02,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [5406c0d4] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:28:02,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-30) [5406c0d4] FINISH, ConnectStorageServerVDSCommand, log id: 43654283 >2014-04-30 02:28:02,704 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-30) [5406c0d4] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:28:02,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 6cf35a79 >2014-04-30 02:28:03,149 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 6cf35a79 >2014-04-30 02:28:03,149 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:28:03,160 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-30) [5bafe351] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:28:03,161 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [5bafe351] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 74086923 >2014-04-30 02:28:03,163 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-30) [5bafe351] FINISH, SetVdsStatusVDSCommand, log id: 74086923 >2014-04-30 02:28:03,172 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [5bafe351] Correlation ID: 5bafe351, Job ID: deabdf76-c38d-4823-9405-02bffd728f70, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:28:03,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [5bafe351] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:28:03,185 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-30) [5bafe351] Correlation ID: 6e4be29f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:28:03,189 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-30) [123c4b37] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:28:03,190 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-30) [7dab0818] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:28:03,191 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-30) [7dab0818] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:30:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-21) [cb00b75] Autorecovering 1 hosts >2014-04-30 02:30:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-21) [cb00b75] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4c372904 >2014-04-30 02:30:00,023 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] FINISH, SetVdsStatusVDSCommand, log id: 4c372904 >2014-04-30 02:30:00,053 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 33085792 >2014-04-30 02:30:00,518 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 36a8a1fc >2014-04-30 02:30:00,740 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] Failed in GetHardwareInfoVDS method >2014-04-30 02:30:00,740 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:30:00,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7922c36c >2014-04-30 02:30:00,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] HostName = 10.34.27.209 >2014-04-30 02:30:00,741 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:30:00,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-21) [59a9babc] FINISH, GetHardwareInfoVDSCommand, log id: 36a8a1fc >2014-04-30 02:30:00,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-21) [59a9babc] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:30:00,758 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-21) [349f00e0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:30:00,759 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-21) [ca31efd] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:30:00,760 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-21) [ca31efd] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 33085792 >2014-04-30 02:30:00,774 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [ca31efd] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:30:00,775 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-21) [ca31efd] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:30:03,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5458a17c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 426f0800 >2014-04-30 02:30:03,786 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5458a17c] Failed in GetHardwareInfoVDS method >2014-04-30 02:30:03,786 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5458a17c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:30:03,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5458a17c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7765a13 >2014-04-30 02:30:03,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5458a17c] HostName = 10.34.27.209 >2014-04-30 02:30:03,786 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5458a17c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:30:03,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5458a17c] FINISH, GetHardwareInfoVDSCommand, log id: 426f0800 >2014-04-30 02:30:03,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [5458a17c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:30:03,801 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [5458a17c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:30:04,022 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-46) [10d7a14f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:30:04,024 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-46) [52bfc36f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:30:04,031 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [52bfc36f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 7c2fd48e >2014-04-30 02:33:04,032 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [52bfc36f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:33:04,032 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [52bfc36f] FINISH, ConnectStorageServerVDSCommand, log id: 7c2fd48e >2014-04-30 02:33:04,032 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-46) [52bfc36f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:33:04,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 6f6ab102 >2014-04-30 02:33:04,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6f6ab102 >2014-04-30 02:33:04,482 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:33:04,491 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-46) [229655cc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:33:04,492 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [229655cc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 548bd90c >2014-04-30 02:33:04,494 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [229655cc] FINISH, SetVdsStatusVDSCommand, log id: 548bd90c >2014-04-30 02:33:04,500 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [229655cc] Correlation ID: 229655cc, Job ID: b5f647d3-ba3a-4803-854e-b2c776654854, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:33:04,506 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [229655cc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:33:04,508 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [229655cc] Correlation ID: 10d7a14f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:33:04,512 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-46) [4d0fb33f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:33:04,513 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-46) [5dd7915] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:33:04,514 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) [5dd7915] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-28) [3f0c5833] Autorecovering 1 hosts >2014-04-30 02:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-28) [3f0c5833] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 48a3eb08 >2014-04-30 02:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] FINISH, SetVdsStatusVDSCommand, log id: 48a3eb08 >2014-04-30 02:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 75ff4a5f >2014-04-30 02:35:00,514 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7027414a >2014-04-30 02:35:00,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] Failed in GetHardwareInfoVDS method >2014-04-30 02:35:00,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:35:00,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@212f9acc >2014-04-30 02:35:00,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] HostName = 10.34.27.209 >2014-04-30 02:35:00,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:35:00,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [41b60445] FINISH, GetHardwareInfoVDSCommand, log id: 7027414a >2014-04-30 02:35:00,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [41b60445] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:35:00,756 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-28) [74fb68dd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:35:00,758 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:35:00,759 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 75ff4a5f >2014-04-30 02:35:00,774 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:35:00,775 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:35:01,872 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [23180e05] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 10967079 >2014-04-30 02:35:02,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [23180e05] Failed in GetHardwareInfoVDS method >2014-04-30 02:35:02,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [23180e05] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:35:02,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [23180e05] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@cb79b10 >2014-04-30 02:35:02,094 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [23180e05] HostName = 10.34.27.209 >2014-04-30 02:35:02,094 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [23180e05] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:35:02,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [23180e05] FINISH, GetHardwareInfoVDSCommand, log id: 10967079 >2014-04-30 02:35:02,099 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [23180e05] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:35:02,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [23180e05] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:35:02,338 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-27) [50a1b406] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:35:02,340 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-27) [66344dcc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:35:02,348 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-27) [66344dcc] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 685efb2a >2014-04-30 02:36:05,017 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-35) [4e705c07] Start refreshing all users data >2014-04-30 02:36:07,296 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3b1dbbaa] Attempting to update VMs/Templates Ovf. >2014-04-30 02:36:07,297 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3b1dbbaa] Attempting to update VM OVFs in Data Center NFS >2014-04-30 02:36:07,298 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3b1dbbaa] Successfully updated VM OVFs in Data Center NFS >2014-04-30 02:36:07,298 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3b1dbbaa] Attempting to update template OVFs in Data Center NFS >2014-04-30 02:36:07,299 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3b1dbbaa] Successfully updated templates OVFs in Data Center NFS >2014-04-30 02:36:07,299 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3b1dbbaa] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-30 02:36:07,299 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-53) [3b1dbbaa] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-30 02:38:02,349 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-27) [66344dcc] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:38:02,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-27) [66344dcc] FINISH, ConnectStorageServerVDSCommand, log id: 685efb2a >2014-04-30 02:38:02,349 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-27) [66344dcc] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:38:02,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 6db6b351 >2014-04-30 02:38:02,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 6db6b351 >2014-04-30 02:38:02,806 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:38:02,817 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-27) [649b9ec4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:38:02,818 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) [649b9ec4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5b4b027a >2014-04-30 02:38:02,820 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) [649b9ec4] FINISH, SetVdsStatusVDSCommand, log id: 5b4b027a >2014-04-30 02:38:02,825 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [649b9ec4] Correlation ID: 649b9ec4, Job ID: 14e61421-83f8-4e7c-b3e1-4310d0ffb9d6, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:38:02,831 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [649b9ec4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:38:02,833 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [649b9ec4] Correlation ID: 50a1b406, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:38:02,839 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-27) [1850dd8f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:38:02,840 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-27) [5ee91683] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:38:02,841 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) [5ee91683] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:40:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [2f82d73b] Autorecovering 1 hosts >2014-04-30 02:40:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [2f82d73b] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:40:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:40:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:40:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:40:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:40:00,024 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 505f1908 >2014-04-30 02:40:00,032 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] FINISH, SetVdsStatusVDSCommand, log id: 505f1908 >2014-04-30 02:40:00,034 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 353e0d39 >2014-04-30 02:40:00,498 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: ffaa78d >2014-04-30 02:40:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] Failed in GetHardwareInfoVDS method >2014-04-30 02:40:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:40:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@110e222d >2014-04-30 02:40:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] HostName = 10.34.27.209 >2014-04-30 02:40:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:40:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [20dd614d] FINISH, GetHardwareInfoVDSCommand, log id: ffaa78d >2014-04-30 02:40:00,725 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [20dd614d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:40:00,736 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-65) [30eb82d3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:40:00,737 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-65) [74644392] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:40:00,738 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) [74644392] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 353e0d39 >2014-04-30 02:40:00,752 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [74644392] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:40:00,753 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [74644392] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:40:03,199 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 34bf8b92 >2014-04-30 02:40:03,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] Failed in GetHardwareInfoVDS method >2014-04-30 02:40:03,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:40:03,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@b64394c >2014-04-30 02:40:03,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] HostName = 10.34.27.209 >2014-04-30 02:40:03,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:40:03,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [30f58968] FINISH, GetHardwareInfoVDSCommand, log id: 34bf8b92 >2014-04-30 02:40:03,427 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [30f58968] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:40:03,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [30f58968] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:40:03,658 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-8) [67a8f085] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:40:03,659 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [16e21c6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:40:03,665 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [16e21c6] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 54d5791b >2014-04-30 02:43:03,667 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [16e21c6] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:43:03,667 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-8) [16e21c6] FINISH, ConnectStorageServerVDSCommand, log id: 54d5791b >2014-04-30 02:43:03,667 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-8) [16e21c6] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:43:03,672 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 12e283fa >2014-04-30 02:43:04,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 12e283fa >2014-04-30 02:43:04,121 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:43:04,132 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-8) [33c4b8eb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:43:04,133 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [33c4b8eb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: b05e84b >2014-04-30 02:43:04,135 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [33c4b8eb] FINISH, SetVdsStatusVDSCommand, log id: b05e84b >2014-04-30 02:43:04,141 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [33c4b8eb] Correlation ID: 33c4b8eb, Job ID: b3dee50f-c2a2-4062-b298-5ae7d1f7d550, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:43:04,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [33c4b8eb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:43:04,150 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [33c4b8eb] Correlation ID: 67a8f085, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:43:04,153 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [522c05b4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:43:04,154 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [1718c2d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:43:04,155 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-8) [1718c2d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:45:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-83) [5473ac4a] Autorecovering 1 hosts >2014-04-30 02:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-83) [5473ac4a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4c77d007 >2014-04-30 02:45:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] FINISH, SetVdsStatusVDSCommand, log id: 4c77d007 >2014-04-30 02:45:00,018 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2c3f0d96 >2014-04-30 02:45:00,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 30922262 >2014-04-30 02:45:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] Failed in GetHardwareInfoVDS method >2014-04-30 02:45:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:45:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4a0f68d7 >2014-04-30 02:45:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] HostName = 10.34.27.209 >2014-04-30 02:45:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:45:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-83) [28cf2ced] FINISH, GetHardwareInfoVDSCommand, log id: 30922262 >2014-04-30 02:45:00,708 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-83) [28cf2ced] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:45:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-83) [5a4a4ffe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:45:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-83) [ff06995] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:45:00,717 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-83) [ff06995] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2c3f0d96 >2014-04-30 02:45:00,732 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [ff06995] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:45:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-83) [ff06995] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:45:01,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 63d59f91 >2014-04-30 02:45:01,730 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] Failed in GetHardwareInfoVDS method >2014-04-30 02:45:01,730 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:45:01,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5ddf1bcb >2014-04-30 02:45:01,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] HostName = 10.34.27.209 >2014-04-30 02:45:01,730 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:45:01,730 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-9) [9a2759] FINISH, GetHardwareInfoVDSCommand, log id: 63d59f91 >2014-04-30 02:45:01,735 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [9a2759] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:45:01,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [9a2759] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:45:01,966 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-9) [3d8bbf68] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:45:01,968 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [220268dd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:45:01,975 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [220268dd] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4cbad031 >2014-04-30 02:48:01,977 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [220268dd] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:48:01,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-9) [220268dd] FINISH, ConnectStorageServerVDSCommand, log id: 4cbad031 >2014-04-30 02:48:01,977 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-9) [220268dd] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:48:01,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 209d4791 >2014-04-30 02:48:02,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 209d4791 >2014-04-30 02:48:02,422 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:48:02,433 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-9) [3ba2efb8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:48:02,434 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-9) [3ba2efb8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 33df92a6 >2014-04-30 02:48:02,436 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-9) [3ba2efb8] FINISH, SetVdsStatusVDSCommand, log id: 33df92a6 >2014-04-30 02:48:02,441 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [3ba2efb8] Correlation ID: 3ba2efb8, Job ID: 191b6fe4-20e0-4741-93e6-0f95122a6280, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:48:02,452 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [3ba2efb8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:48:02,454 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-9) [3ba2efb8] Correlation ID: 3d8bbf68, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:48:02,459 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-9) [4277fea6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:48:02,460 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-9) [4d79f29e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:48:02,461 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-9) [4d79f29e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) [2bd3cf05] Autorecovering 1 hosts >2014-04-30 02:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) [2bd3cf05] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 507e1109 >2014-04-30 02:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] FINISH, SetVdsStatusVDSCommand, log id: 507e1109 >2014-04-30 02:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 122f9bc6 >2014-04-30 02:50:00,522 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1872db13 >2014-04-30 02:50:00,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] Failed in GetHardwareInfoVDS method >2014-04-30 02:50:00,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:50:00,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1c8cc8ef >2014-04-30 02:50:00,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] HostName = 10.34.27.209 >2014-04-30 02:50:00,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:50:00,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] FINISH, GetHardwareInfoVDSCommand, log id: 1872db13 >2014-04-30 02:50:00,751 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [3a7f1e7f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:50:00,761 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-54) [9d20de1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:50:00,762 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-54) [43b2e52a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:50:00,763 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) [43b2e52a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 122f9bc6 >2014-04-30 02:50:00,778 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [43b2e52a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:50:00,779 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [43b2e52a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:50:02,836 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [ea7eac6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 35263ac8 >2014-04-30 02:50:03,059 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [ea7eac6] Failed in GetHardwareInfoVDS method >2014-04-30 02:50:03,059 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [ea7eac6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:50:03,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [ea7eac6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6544a24f >2014-04-30 02:50:03,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [ea7eac6] HostName = 10.34.27.209 >2014-04-30 02:50:03,059 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [ea7eac6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:50:03,059 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [ea7eac6] FINISH, GetHardwareInfoVDSCommand, log id: 35263ac8 >2014-04-30 02:50:03,064 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [ea7eac6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:50:03,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [ea7eac6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:50:03,297 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [294c46a8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:50:03,299 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [7ac62c20] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:50:03,305 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [7ac62c20] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1d8d54af >2014-04-30 02:53:03,306 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [7ac62c20] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:53:03,306 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [7ac62c20] FINISH, ConnectStorageServerVDSCommand, log id: 1d8d54af >2014-04-30 02:53:03,306 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [7ac62c20] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:53:03,310 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5c1868d4 >2014-04-30 02:53:03,752 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5c1868d4 >2014-04-30 02:53:03,752 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:53:03,763 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-23) [110c758d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:53:03,764 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [110c758d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 583ebd89 >2014-04-30 02:53:03,767 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [110c758d] FINISH, SetVdsStatusVDSCommand, log id: 583ebd89 >2014-04-30 02:53:03,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [110c758d] Correlation ID: 110c758d, Job ID: b7329573-4409-4ae2-b199-d540cf716eb7, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:53:03,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [110c758d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:53:03,811 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [110c758d] Correlation ID: 294c46a8, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:53:03,816 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-23) [43d18c7e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:53:03,817 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-23) [7327d517] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:53:03,817 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) [7327d517] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 02:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-52) [897e89b] Autorecovering 1 hosts >2014-04-30 02:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-52) [897e89b] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 02:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 02:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 02:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 458115a2 >2014-04-30 02:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] FINISH, SetVdsStatusVDSCommand, log id: 458115a2 >2014-04-30 02:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 56e91b90 >2014-04-30 02:55:00,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 81fb7f8 >2014-04-30 02:55:00,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] Failed in GetHardwareInfoVDS method >2014-04-30 02:55:00,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:55:00,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5a66fef2 >2014-04-30 02:55:00,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] HostName = 10.34.27.209 >2014-04-30 02:55:00,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:55:00,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [3f4c484f] FINISH, GetHardwareInfoVDSCommand, log id: 81fb7f8 >2014-04-30 02:55:00,730 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [3f4c484f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:55:00,742 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-52) [693d773e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:55:00,744 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:55:00,745 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 56e91b90 >2014-04-30 02:55:00,763 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 02:55:00,764 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 02:55:01,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2ba94504 >2014-04-30 02:55:01,422 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] Failed in GetHardwareInfoVDS method >2014-04-30 02:55:01,423 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:55:01,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@8abc583 >2014-04-30 02:55:01,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] HostName = 10.34.27.209 >2014-04-30 02:55:01,423 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 02:55:01,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] FINISH, GetHardwareInfoVDSCommand, log id: 2ba94504 >2014-04-30 02:55:01,427 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 02:55:01,436 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [51fa3fd7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 02:55:01,689 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-77) [2bf0cfc7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:55:01,691 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [6163e416] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 02:55:01,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [6163e416] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: dbb9bed >2014-04-30 02:58:01,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [6163e416] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 02:58:01,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [6163e416] FINISH, ConnectStorageServerVDSCommand, log id: dbb9bed >2014-04-30 02:58:01,699 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [6163e416] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 02:58:01,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1fe9d27a >2014-04-30 02:58:02,168 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 1fe9d27a >2014-04-30 02:58:02,168 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 02:58:02,179 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-77) [351f570f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:58:02,180 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [351f570f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7fd0d752 >2014-04-30 02:58:02,182 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [351f570f] FINISH, SetVdsStatusVDSCommand, log id: 7fd0d752 >2014-04-30 02:58:02,188 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [351f570f] Correlation ID: 351f570f, Job ID: b4cd676a-0360-4363-8dd6-73715f83179c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 02:58:02,199 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [351f570f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 02:58:02,201 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [351f570f] Correlation ID: 2bf0cfc7, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 02:58:02,205 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [2da5829f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:58:02,206 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [34a171a7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 02:58:02,206 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [34a171a7] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [17ec17ae] Autorecovering 1 hosts >2014-04-30 03:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [17ec17ae] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7ae4067c >2014-04-30 03:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] FINISH, SetVdsStatusVDSCommand, log id: 7ae4067c >2014-04-30 03:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 44fd599d >2014-04-30 03:00:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 506aeb14 >2014-04-30 03:00:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] Failed in GetHardwareInfoVDS method >2014-04-30 03:00:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:00:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2f878b39 >2014-04-30 03:00:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] HostName = 10.34.27.209 >2014-04-30 03:00:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:00:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [7a080b9e] FINISH, GetHardwareInfoVDSCommand, log id: 506aeb14 >2014-04-30 03:00:00,709 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [7a080b9e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:00:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [3ececb95] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:00:00,726 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [4e9fed15] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:00:00,727 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [4e9fed15] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 44fd599d >2014-04-30 03:00:00,741 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [4e9fed15] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:00:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [4e9fed15] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:00:02,649 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6012f4e2 >2014-04-30 03:00:02,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] Failed in GetHardwareInfoVDS method >2014-04-30 03:00:02,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:00:02,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7c8b52b4 >2014-04-30 03:00:02,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] HostName = 10.34.27.209 >2014-04-30 03:00:02,870 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:00:02,870 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-37) [5fa14f3] FINISH, GetHardwareInfoVDSCommand, log id: 6012f4e2 >2014-04-30 03:00:02,875 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [5fa14f3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:00:02,883 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [5fa14f3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:00:03,105 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-37) [11cbedec] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:00:03,106 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-37) [507bfc1a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:00:03,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [507bfc1a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 71da1c2 >2014-04-30 03:03:03,114 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [507bfc1a] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:03:03,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-37) [507bfc1a] FINISH, ConnectStorageServerVDSCommand, log id: 71da1c2 >2014-04-30 03:03:03,114 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-37) [507bfc1a] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:03:03,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 49e36693 >2014-04-30 03:03:03,563 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 49e36693 >2014-04-30 03:03:03,563 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:03:03,574 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-37) [173c3d6b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:03:03,575 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [173c3d6b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1008a12a >2014-04-30 03:03:03,577 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-37) [173c3d6b] FINISH, SetVdsStatusVDSCommand, log id: 1008a12a >2014-04-30 03:03:03,583 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [173c3d6b] Correlation ID: 173c3d6b, Job ID: faa0180a-c3a1-43d6-a1ee-9cd9ab597867, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:03:03,590 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [173c3d6b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:03:03,592 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-37) [173c3d6b] Correlation ID: 11cbedec, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:03:03,595 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-37) [5fc5761e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:03:03,597 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-37) [7311b8c4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:03:03,597 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-37) [7311b8c4] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:05:00,014 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-70) [67291b24] Autorecovering 1 hosts >2014-04-30 03:05:00,014 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-70) [67291b24] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:05:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [43857511] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:05:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [43857511] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:05:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [43857511] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:05:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [43857511] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:05:00,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [43857511] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4a8ff8e5 >2014-04-30 03:05:00,026 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [43857511] FINISH, SetVdsStatusVDSCommand, log id: 4a8ff8e5 >2014-04-30 03:05:00,027 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) [43857511] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2a977d2c >2014-04-30 03:05:00,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [43857511] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 31da19bd >2014-04-30 03:05:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [43857511] Failed in GetHardwareInfoVDS method >2014-04-30 03:05:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [43857511] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:05:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [43857511] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@65cfe0dc >2014-04-30 03:05:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [43857511] HostName = 10.34.27.209 >2014-04-30 03:05:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [43857511] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:05:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [43857511] FINISH, GetHardwareInfoVDSCommand, log id: 31da19bd >2014-04-30 03:05:00,723 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [43857511] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:05:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-70) [1381a0e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:05:00,732 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-70) [69148df6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:05:00,732 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) [69148df6] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2a977d2c >2014-04-30 03:05:00,747 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [69148df6] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:05:00,748 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [69148df6] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:05:03,968 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3704d5e7 >2014-04-30 03:05:04,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] Failed in GetHardwareInfoVDS method >2014-04-30 03:05:04,189 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:05:04,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1e774ce9 >2014-04-30 03:05:04,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] HostName = 10.34.27.209 >2014-04-30 03:05:04,190 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:05:04,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [49478d71] FINISH, GetHardwareInfoVDSCommand, log id: 3704d5e7 >2014-04-30 03:05:04,195 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [49478d71] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:05:04,239 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [49478d71] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:05:04,461 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-22) [4a663baa] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:05:04,462 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [252ca941] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:05:04,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [252ca941] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 28771460 >2014-04-30 03:08:04,470 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [252ca941] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:08:04,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [252ca941] FINISH, ConnectStorageServerVDSCommand, log id: 28771460 >2014-04-30 03:08:04,470 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [252ca941] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:08:04,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 12f20a0b >2014-04-30 03:08:04,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 12f20a0b >2014-04-30 03:08:04,922 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:08:04,933 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-22) [16968b17] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:08:04,934 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [16968b17] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 79ffb0cb >2014-04-30 03:08:04,937 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [16968b17] FINISH, SetVdsStatusVDSCommand, log id: 79ffb0cb >2014-04-30 03:08:04,943 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [16968b17] Correlation ID: 16968b17, Job ID: ef47f582-e6b6-49c4-b987-d918df98d6fd, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:08:04,950 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [16968b17] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:08:04,952 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [16968b17] Correlation ID: 4a663baa, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:08:04,957 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-22) [702e4c0f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:08:04,958 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-22) [7337d50d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:08:04,959 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [7337d50d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-18) [4fd8ab38] Autorecovering 1 hosts >2014-04-30 03:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-18) [4fd8ab38] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 70f3c3a9 >2014-04-30 03:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] FINISH, SetVdsStatusVDSCommand, log id: 70f3c3a9 >2014-04-30 03:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1937d9db >2014-04-30 03:10:00,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 34f8f49b >2014-04-30 03:10:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] Failed in GetHardwareInfoVDS method >2014-04-30 03:10:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:10:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@44830748 >2014-04-30 03:10:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] HostName = 10.34.27.209 >2014-04-30 03:10:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:10:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [4411c1bd] FINISH, GetHardwareInfoVDSCommand, log id: 34f8f49b >2014-04-30 03:10:00,719 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [4411c1bd] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:10:00,763 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-18) [62023027] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:10:00,764 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:10:00,765 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1937d9db >2014-04-30 03:10:00,780 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:10:00,781 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:10:02,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 69c52cad >2014-04-30 03:10:02,538 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] Failed in GetHardwareInfoVDS method >2014-04-30 03:10:02,538 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:10:02,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3cc325ae >2014-04-30 03:10:02,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] HostName = 10.34.27.209 >2014-04-30 03:10:02,538 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:10:02,538 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4e705c07] FINISH, GetHardwareInfoVDSCommand, log id: 69c52cad >2014-04-30 03:10:02,542 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [4e705c07] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:10:02,551 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [4e705c07] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:10:02,772 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-35) [29b0d0ab] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:10:02,774 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-35) [583c037e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:10:02,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-35) [583c037e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1b4be001 >2014-04-30 03:13:02,783 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-35) [583c037e] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:13:02,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-35) [583c037e] FINISH, ConnectStorageServerVDSCommand, log id: 1b4be001 >2014-04-30 03:13:02,783 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-35) [583c037e] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:13:02,786 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 22987673 >2014-04-30 03:13:03,235 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 22987673 >2014-04-30 03:13:03,235 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:13:03,249 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-35) [d4b97a6] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:13:03,250 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [d4b97a6] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5cf53f34 >2014-04-30 03:13:03,253 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [d4b97a6] FINISH, SetVdsStatusVDSCommand, log id: 5cf53f34 >2014-04-30 03:13:03,260 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [d4b97a6] Correlation ID: d4b97a6, Job ID: 74d2b45d-428b-4de0-9801-63d98ee4b7e5, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:13:03,269 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [d4b97a6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:13:03,271 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [d4b97a6] Correlation ID: 29b0d0ab, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:13:03,277 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-35) [42fbb801] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:13:03,278 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-35) [3b5a09da] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:13:03,279 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-35) [3b5a09da] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-100) [169df275] Autorecovering 1 hosts >2014-04-30 03:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-100) [169df275] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 578fad8 >2014-04-30 03:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] FINISH, SetVdsStatusVDSCommand, log id: 578fad8 >2014-04-30 03:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 28cc4f5d >2014-04-30 03:15:00,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3ce7b40d >2014-04-30 03:15:00,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] Failed in GetHardwareInfoVDS method >2014-04-30 03:15:00,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:15:00,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@fdde0b8 >2014-04-30 03:15:00,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] HostName = 10.34.27.209 >2014-04-30 03:15:00,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:15:00,701 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-100) [7724191e] FINISH, GetHardwareInfoVDSCommand, log id: 3ce7b40d >2014-04-30 03:15:00,707 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-100) [7724191e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:15:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-100) [705c9d58] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:15:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-100) [2952d560] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:15:00,717 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-100) [2952d560] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 28cc4f5d >2014-04-30 03:15:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [2952d560] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:15:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-100) [2952d560] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:15:03,628 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [2126fd16] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: d1895d5 >2014-04-30 03:15:03,851 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [2126fd16] Failed in GetHardwareInfoVDS method >2014-04-30 03:15:03,851 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [2126fd16] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:15:03,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [2126fd16] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4914d9aa >2014-04-30 03:15:03,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [2126fd16] HostName = 10.34.27.209 >2014-04-30 03:15:03,851 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [2126fd16] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:15:03,851 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [2126fd16] FINISH, GetHardwareInfoVDSCommand, log id: d1895d5 >2014-04-30 03:15:03,856 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [2126fd16] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:15:03,865 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [2126fd16] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:15:04,087 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-93) [bc5f6a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:15:04,089 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [74f82a16] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:15:04,097 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [74f82a16] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2cb999a3 >2014-04-30 03:18:04,099 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [74f82a16] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:18:04,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [74f82a16] FINISH, ConnectStorageServerVDSCommand, log id: 2cb999a3 >2014-04-30 03:18:04,100 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [74f82a16] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:18:04,114 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 77ec1a1e >2014-04-30 03:18:04,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 77ec1a1e >2014-04-30 03:18:04,570 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:18:04,584 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-93) [382c4c04] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:18:04,585 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [382c4c04] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 372d8bd9 >2014-04-30 03:18:04,588 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [382c4c04] FINISH, SetVdsStatusVDSCommand, log id: 372d8bd9 >2014-04-30 03:18:04,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [382c4c04] Correlation ID: 382c4c04, Job ID: 541ee93b-7740-4165-82ec-e77a1063fc4f, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:18:04,602 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [382c4c04] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:18:04,604 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [382c4c04] Correlation ID: bc5f6a, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:18:04,608 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [2680055c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:18:04,609 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [4f5a836] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:18:04,610 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [4f5a836] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-63) [2af8f9ae] Autorecovering 1 hosts >2014-04-30 03:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-63) [2af8f9ae] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 25070f73 >2014-04-30 03:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] FINISH, SetVdsStatusVDSCommand, log id: 25070f73 >2014-04-30 03:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3ca5743e >2014-04-30 03:20:00,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 31d7e00c >2014-04-30 03:20:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] Failed in GetHardwareInfoVDS method >2014-04-30 03:20:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:20:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@343a37d >2014-04-30 03:20:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] HostName = 10.34.27.209 >2014-04-30 03:20:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:20:00,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [d147bf2] FINISH, GetHardwareInfoVDSCommand, log id: 31d7e00c >2014-04-30 03:20:00,708 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [d147bf2] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:20:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-63) [3bd892a9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:20:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:20:00,718 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3ca5743e >2014-04-30 03:20:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:20:00,732 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:20:01,965 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [34a171a7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 47a44766 >2014-04-30 03:20:02,188 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [34a171a7] Failed in GetHardwareInfoVDS method >2014-04-30 03:20:02,188 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [34a171a7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:20:02,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [34a171a7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2e61feac >2014-04-30 03:20:02,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [34a171a7] HostName = 10.34.27.209 >2014-04-30 03:20:02,188 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [34a171a7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:20:02,188 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [34a171a7] FINISH, GetHardwareInfoVDSCommand, log id: 47a44766 >2014-04-30 03:20:02,193 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [34a171a7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:20:02,202 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [34a171a7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:20:02,425 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-77) [3925d711] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:20:02,426 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [513e25ad] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:20:02,432 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [513e25ad] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 420bea6b >2014-04-30 03:23:02,434 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [513e25ad] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:23:02,434 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-77) [513e25ad] FINISH, ConnectStorageServerVDSCommand, log id: 420bea6b >2014-04-30 03:23:02,434 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-77) [513e25ad] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:23:02,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 25127436 >2014-04-30 03:23:02,878 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 25127436 >2014-04-30 03:23:02,878 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:23:02,889 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-77) [5424927e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:23:02,891 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [5424927e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 22575363 >2014-04-30 03:23:02,893 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [5424927e] FINISH, SetVdsStatusVDSCommand, log id: 22575363 >2014-04-30 03:23:02,899 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [5424927e] Correlation ID: 5424927e, Job ID: 67518d0d-a434-4024-937f-61768c2a989e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:23:02,905 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [5424927e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:23:02,907 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [5424927e] Correlation ID: 3925d711, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:23:02,913 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [1ec20481] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:23:02,914 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [1b5b347b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:23:02,915 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-77) [1b5b347b] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-42) [29340fe2] Autorecovering 1 hosts >2014-04-30 03:25:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-42) [29340fe2] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:25:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:25:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6fd5b05 >2014-04-30 03:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] FINISH, SetVdsStatusVDSCommand, log id: 6fd5b05 >2014-04-30 03:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 385cf3f8 >2014-04-30 03:25:00,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6df9a54d >2014-04-30 03:25:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] Failed in GetHardwareInfoVDS method >2014-04-30 03:25:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:25:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@42966700 >2014-04-30 03:25:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] HostName = 10.34.27.209 >2014-04-30 03:25:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:25:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] FINISH, GetHardwareInfoVDSCommand, log id: 6df9a54d >2014-04-30 03:25:00,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [6dea8e6d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:25:00,754 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-42) [5cd19b47] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:25:00,755 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:25:00,761 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 385cf3f8 >2014-04-30 03:25:00,783 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:25:00,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [1770113a] Correlation ID: 6dea8e6d, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was autorecovered. >2014-04-30 03:25:00,786 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:25:03,273 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [5fe11796] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5ad5662f >2014-04-30 03:25:03,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [5fe11796] Failed in GetHardwareInfoVDS method >2014-04-30 03:25:03,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [5fe11796] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:25:03,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [5fe11796] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@18ac8585 >2014-04-30 03:25:03,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [5fe11796] HostName = 10.34.27.209 >2014-04-30 03:25:03,497 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [5fe11796] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:25:03,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-2) [5fe11796] FINISH, GetHardwareInfoVDSCommand, log id: 5ad5662f >2014-04-30 03:25:03,502 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [5fe11796] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:25:03,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [5fe11796] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:25:03,732 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-2) [7eca22a4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:25:03,734 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-2) [24ef2d63] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:25:03,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-2) [24ef2d63] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1397e2a9 >2014-04-30 03:28:03,741 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-2) [24ef2d63] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:28:03,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-2) [24ef2d63] FINISH, ConnectStorageServerVDSCommand, log id: 1397e2a9 >2014-04-30 03:28:03,741 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-2) [24ef2d63] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:28:03,746 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3c4ff702 >2014-04-30 03:28:04,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 3c4ff702 >2014-04-30 03:28:04,190 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:28:04,201 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-2) [4e7656bd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:28:04,202 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [4e7656bd] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4a94417a >2014-04-30 03:28:04,204 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-2) [4e7656bd] FINISH, SetVdsStatusVDSCommand, log id: 4a94417a >2014-04-30 03:28:04,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [4e7656bd] Correlation ID: 4e7656bd, Job ID: 8994ffb2-d6eb-4afd-bf8e-1b33bc9b660c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:28:04,217 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [4e7656bd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:28:04,219 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-2) [4e7656bd] Correlation ID: 7eca22a4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:28:04,223 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-2) [6a749b44] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:28:04,224 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-2) [4e501284] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:28:04,225 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-2) [4e501284] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-70) [69148df6] Autorecovering 1 hosts >2014-04-30 03:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-70) [69148df6] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:30:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:30:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:30:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:30:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5e0ecdf7 >2014-04-30 03:30:00,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] FINISH, SetVdsStatusVDSCommand, log id: 5e0ecdf7 >2014-04-30 03:30:00,019 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 20d218e8 >2014-04-30 03:30:00,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 540ec285 >2014-04-30 03:30:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] Failed in GetHardwareInfoVDS method >2014-04-30 03:30:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:30:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4c20bad2 >2014-04-30 03:30:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] HostName = 10.34.27.209 >2014-04-30 03:30:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:30:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] FINISH, GetHardwareInfoVDSCommand, log id: 540ec285 >2014-04-30 03:30:00,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [1ee06ab2] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:30:00,734 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-70) [2a98b29b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:30:00,735 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:30:00,736 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 20d218e8 >2014-04-30 03:30:00,752 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:30:00,753 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:30:01,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [2c8d8414] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3146c829 >2014-04-30 03:30:01,805 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [2c8d8414] Failed in GetHardwareInfoVDS method >2014-04-30 03:30:01,805 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [2c8d8414] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:30:01,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [2c8d8414] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1dff1505 >2014-04-30 03:30:01,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [2c8d8414] HostName = 10.34.27.209 >2014-04-30 03:30:01,805 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [2c8d8414] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:30:01,805 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [2c8d8414] FINISH, GetHardwareInfoVDSCommand, log id: 3146c829 >2014-04-30 03:30:01,810 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [2c8d8414] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:30:01,820 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [2c8d8414] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:30:02,042 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [9d09a51] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:30:02,043 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [67a656ad] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:30:02,049 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [67a656ad] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 19183b39 >2014-04-30 03:33:02,051 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [67a656ad] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:33:02,051 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [67a656ad] FINISH, ConnectStorageServerVDSCommand, log id: 19183b39 >2014-04-30 03:33:02,051 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [67a656ad] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:33:02,055 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 456ebec9 >2014-04-30 03:33:02,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 456ebec9 >2014-04-30 03:33:02,497 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:33:02,507 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-86) [23121ca0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:33:02,508 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [23121ca0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 678102e2 >2014-04-30 03:33:02,510 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [23121ca0] FINISH, SetVdsStatusVDSCommand, log id: 678102e2 >2014-04-30 03:33:02,516 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [23121ca0] Correlation ID: 23121ca0, Job ID: 431f04ed-6b04-4e40-9e3a-4bed0518713a, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:33:02,522 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [23121ca0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:33:02,524 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [23121ca0] Correlation ID: 9d09a51, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:33:02,528 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [75b122e2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:33:02,529 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [30770c8d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:33:02,530 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [30770c8d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) [803988a] Autorecovering 1 hosts >2014-04-30 03:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-50) [803988a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5436c40b >2014-04-30 03:35:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] FINISH, SetVdsStatusVDSCommand, log id: 5436c40b >2014-04-30 03:35:00,019 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 324bf691 >2014-04-30 03:35:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7c40737f >2014-04-30 03:35:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] Failed in GetHardwareInfoVDS method >2014-04-30 03:35:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:35:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@fc4da1e >2014-04-30 03:35:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] HostName = 10.34.27.209 >2014-04-30 03:35:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:35:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2aa98167] FINISH, GetHardwareInfoVDSCommand, log id: 7c40737f >2014-04-30 03:35:00,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [2aa98167] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:35:00,727 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [24c29567] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:35:00,728 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:35:00,729 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 324bf691 >2014-04-30 03:35:00,743 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:35:00,744 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:35:02,915 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 662f810c >2014-04-30 03:35:03,141 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] Failed in GetHardwareInfoVDS method >2014-04-30 03:35:03,141 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:35:03,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@38c16f3d >2014-04-30 03:35:03,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] HostName = 10.34.27.209 >2014-04-30 03:35:03,142 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:35:03,142 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [3875d836] FINISH, GetHardwareInfoVDSCommand, log id: 662f810c >2014-04-30 03:35:03,234 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [3875d836] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:35:03,247 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [3875d836] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:35:03,470 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [44092750] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:35:03,472 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [234aeda0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:35:03,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [234aeda0] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4bcdb3b >2014-04-30 03:35:35,001 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-98) [48e5821f] Start deleteAgedOutAuditLogs >2014-04-30 03:35:35,003 INFO [org.ovirt.engine.core.bll.AuditLogCleanupManager] (DefaultQuartzScheduler_Worker-98) [48e5821f] Finished deleteAgedOutAuditLogs >2014-04-30 03:36:05,019 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] Start refreshing all users data >2014-04-30 03:36:07,300 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [2952d560] Attempting to update VMs/Templates Ovf. >2014-04-30 03:36:07,301 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [2952d560] Attempting to update VM OVFs in Data Center NFS >2014-04-30 03:36:07,302 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [2952d560] Successfully updated VM OVFs in Data Center NFS >2014-04-30 03:36:07,302 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [2952d560] Attempting to update template OVFs in Data Center NFS >2014-04-30 03:36:07,302 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [2952d560] Successfully updated templates OVFs in Data Center NFS >2014-04-30 03:36:07,302 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [2952d560] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-30 03:36:07,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-100) [2952d560] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-30 03:38:03,480 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [234aeda0] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:38:03,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [234aeda0] FINISH, ConnectStorageServerVDSCommand, log id: 4bcdb3b >2014-04-30 03:38:03,480 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [234aeda0] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:38:03,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 201aac8e >2014-04-30 03:38:03,933 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 201aac8e >2014-04-30 03:38:03,933 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:38:03,944 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-84) [4300b00d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:38:03,946 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [4300b00d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6a729b9b >2014-04-30 03:38:03,948 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [4300b00d] FINISH, SetVdsStatusVDSCommand, log id: 6a729b9b >2014-04-30 03:38:03,955 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [4300b00d] Correlation ID: 4300b00d, Job ID: cc51a898-929e-487e-a0e9-7dd2af671a56, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:38:03,962 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [4300b00d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:38:03,964 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [4300b00d] Correlation ID: 44092750, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:38:03,969 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [3e836bf9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:38:03,970 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [48585b8e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:38:03,971 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) [48585b8e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-4) [316204bc] Autorecovering 1 hosts >2014-04-30 03:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-4) [316204bc] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2e92f07 >2014-04-30 03:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] FINISH, SetVdsStatusVDSCommand, log id: 2e92f07 >2014-04-30 03:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6913c5b0 >2014-04-30 03:40:00,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7689fed7 >2014-04-30 03:40:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] Failed in GetHardwareInfoVDS method >2014-04-30 03:40:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:40:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4f71526 >2014-04-30 03:40:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] HostName = 10.34.27.209 >2014-04-30 03:40:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:40:00,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [1cb42951] FINISH, GetHardwareInfoVDSCommand, log id: 7689fed7 >2014-04-30 03:40:00,700 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [1cb42951] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:40:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-4) [6b7db51f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:40:00,710 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:40:00,711 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6913c5b0 >2014-04-30 03:40:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:40:00,734 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:40:01,328 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [522d21f3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 44d75ff6 >2014-04-30 03:40:01,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [522d21f3] Failed in GetHardwareInfoVDS method >2014-04-30 03:40:01,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [522d21f3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:40:01,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [522d21f3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@17dd32c9 >2014-04-30 03:40:01,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [522d21f3] HostName = 10.34.27.209 >2014-04-30 03:40:01,550 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [522d21f3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:40:01,550 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [522d21f3] FINISH, GetHardwareInfoVDSCommand, log id: 44d75ff6 >2014-04-30 03:40:01,555 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [522d21f3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:40:01,563 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [522d21f3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:40:01,786 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-19) [42cb7025] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:40:01,787 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [45b894f6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:40:01,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [45b894f6] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 639b9751 >2014-04-30 03:43:01,795 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [45b894f6] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:43:01,795 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [45b894f6] FINISH, ConnectStorageServerVDSCommand, log id: 639b9751 >2014-04-30 03:43:01,795 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [45b894f6] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:43:01,799 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 185b81a8 >2014-04-30 03:43:02,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 185b81a8 >2014-04-30 03:43:02,254 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:43:02,265 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-19) [5e2a9d69] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:43:02,266 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [5e2a9d69] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3327c08e >2014-04-30 03:43:02,268 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [5e2a9d69] FINISH, SetVdsStatusVDSCommand, log id: 3327c08e >2014-04-30 03:43:02,274 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [5e2a9d69] Correlation ID: 5e2a9d69, Job ID: a859b99e-288d-4012-866b-5f122b82fbea, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:43:02,281 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [5e2a9d69] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:43:02,284 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [5e2a9d69] Correlation ID: 42cb7025, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:43:02,288 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-19) [249b914f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:43:02,289 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-19) [7ffad043] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:43:02,290 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [7ffad043] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) [7fa0aac6] Autorecovering 1 hosts >2014-04-30 03:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-48) [7fa0aac6] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:45:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 182b759b >2014-04-30 03:45:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] FINISH, SetVdsStatusVDSCommand, log id: 182b759b >2014-04-30 03:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6dcc2b73 >2014-04-30 03:45:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 54039ee6 >2014-04-30 03:45:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] Failed in GetHardwareInfoVDS method >2014-04-30 03:45:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:45:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@39b53921 >2014-04-30 03:45:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] HostName = 10.34.27.209 >2014-04-30 03:45:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:45:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [6146fc0a] FINISH, GetHardwareInfoVDSCommand, log id: 54039ee6 >2014-04-30 03:45:00,939 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [6146fc0a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:45:00,948 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [573c1ff6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:45:00,949 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:45:00,950 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6dcc2b73 >2014-04-30 03:45:00,963 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:45:00,964 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:45:02,651 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: e5e251 >2014-04-30 03:45:02,873 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] Failed in GetHardwareInfoVDS method >2014-04-30 03:45:02,873 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:45:02,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@746d344c >2014-04-30 03:45:02,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] HostName = 10.34.27.209 >2014-04-30 03:45:02,873 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:45:02,873 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [52a05563] FINISH, GetHardwareInfoVDSCommand, log id: e5e251 >2014-04-30 03:45:03,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [52a05563] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:45:03,116 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [52a05563] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:45:03,339 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [1c75a070] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:45:03,341 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [5c43e60] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:45:03,347 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [5c43e60] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 67e4187e >2014-04-30 03:48:03,349 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [5c43e60] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:48:03,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [5c43e60] FINISH, ConnectStorageServerVDSCommand, log id: 67e4187e >2014-04-30 03:48:03,349 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [5c43e60] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:48:03,353 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 29d97afc >2014-04-30 03:48:03,812 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 29d97afc >2014-04-30 03:48:03,812 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:48:03,825 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-81) [2ecfba94] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:48:03,826 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [2ecfba94] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 172b1eec >2014-04-30 03:48:03,829 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [2ecfba94] FINISH, SetVdsStatusVDSCommand, log id: 172b1eec >2014-04-30 03:48:03,836 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [2ecfba94] Correlation ID: 2ecfba94, Job ID: 855fcd3a-73d0-4707-b5d5-2f4819c4f660, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:48:03,842 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [2ecfba94] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:48:03,846 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [2ecfba94] Correlation ID: 1c75a070, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:48:03,850 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [1124558b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:48:03,851 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [51c4293] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:48:03,852 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [51c4293] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-76) [44235e17] Autorecovering 1 hosts >2014-04-30 03:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-76) [44235e17] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6d56b237 >2014-04-30 03:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] FINISH, SetVdsStatusVDSCommand, log id: 6d56b237 >2014-04-30 03:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 604b2afb >2014-04-30 03:50:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4ab95a9a >2014-04-30 03:50:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] Failed in GetHardwareInfoVDS method >2014-04-30 03:50:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:50:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2bb1118a >2014-04-30 03:50:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] HostName = 10.34.27.209 >2014-04-30 03:50:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:50:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] FINISH, GetHardwareInfoVDSCommand, log id: 4ab95a9a >2014-04-30 03:50:00,705 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [aa4bdfa] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:50:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-76) [729244a7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:50:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-76) [3bd66c3a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:50:00,717 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [3bd66c3a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 604b2afb >2014-04-30 03:50:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [3bd66c3a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:50:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [3bd66c3a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:50:01,212 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 49154f52 >2014-04-30 03:50:01,447 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] Failed in GetHardwareInfoVDS method >2014-04-30 03:50:01,447 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:50:01,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@160ab5fa >2014-04-30 03:50:01,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] HostName = 10.34.27.209 >2014-04-30 03:50:01,447 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:50:01,447 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] FINISH, GetHardwareInfoVDSCommand, log id: 49154f52 >2014-04-30 03:50:01,452 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:50:01,463 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [48fe6cd5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:50:01,732 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-5) [1fe5c694] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:50:01,734 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-5) [3869ccc1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:50:01,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-5) [3869ccc1] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 36643fdc >2014-04-30 03:53:01,741 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-5) [3869ccc1] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:53:01,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-5) [3869ccc1] FINISH, ConnectStorageServerVDSCommand, log id: 36643fdc >2014-04-30 03:53:01,741 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-5) [3869ccc1] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:53:01,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7487da0d >2014-04-30 03:53:02,185 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7487da0d >2014-04-30 03:53:02,185 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:53:02,197 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-5) [6b455ac0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:53:02,198 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [6b455ac0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2f071c9c >2014-04-30 03:53:02,201 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [6b455ac0] FINISH, SetVdsStatusVDSCommand, log id: 2f071c9c >2014-04-30 03:53:02,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [6b455ac0] Correlation ID: 6b455ac0, Job ID: e052a52f-c640-47d7-8c2d-576a0bcc3909, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:53:02,220 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [6b455ac0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:53:02,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [6b455ac0] Correlation ID: 1fe5c694, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:53:02,244 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-5) [101fd16a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:53:02,245 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-5) [3e608cce] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:53:02,246 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [3e608cce] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 03:55:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [65b46d31] Autorecovering 1 hosts >2014-04-30 03:55:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [65b46d31] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 03:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 03:55:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 03:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3ac84557 >2014-04-30 03:55:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] FINISH, SetVdsStatusVDSCommand, log id: 3ac84557 >2014-04-30 03:55:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 62ffba34 >2014-04-30 03:55:00,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 36b58559 >2014-04-30 03:55:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] Failed in GetHardwareInfoVDS method >2014-04-30 03:55:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:55:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4745b21e >2014-04-30 03:55:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] HostName = 10.34.27.209 >2014-04-30 03:55:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:55:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [23b1a643] FINISH, GetHardwareInfoVDSCommand, log id: 36b58559 >2014-04-30 03:55:00,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [23b1a643] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:55:00,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-56) [23333aa8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:55:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-56) [46147fed] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:55:00,732 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [46147fed] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 62ffba34 >2014-04-30 03:55:00,751 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [46147fed] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 03:55:00,752 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [46147fed] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 03:55:02,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [7743b53d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4bed2aa9 >2014-04-30 03:55:02,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [7743b53d] Failed in GetHardwareInfoVDS method >2014-04-30 03:55:02,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [7743b53d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:55:02,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [7743b53d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3f8be430 >2014-04-30 03:55:02,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [7743b53d] HostName = 10.34.27.209 >2014-04-30 03:55:02,854 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [7743b53d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 03:55:02,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [7743b53d] FINISH, GetHardwareInfoVDSCommand, log id: 4bed2aa9 >2014-04-30 03:55:02,862 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [7743b53d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 03:55:02,890 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [7743b53d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 03:55:03,111 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-32) [4ee48c09] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:55:03,113 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-32) [4c664920] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 03:55:03,119 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [4c664920] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 13d17c3b >2014-04-30 03:58:03,121 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [4c664920] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 03:58:03,121 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [4c664920] FINISH, ConnectStorageServerVDSCommand, log id: 13d17c3b >2014-04-30 03:58:03,121 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-32) [4c664920] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 03:58:03,124 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 6edcd20b >2014-04-30 03:58:03,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6edcd20b >2014-04-30 03:58:03,564 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 03:58:03,580 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-32) [3cb2d7c5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:58:03,581 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [3cb2d7c5] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5144548a >2014-04-30 03:58:03,584 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [3cb2d7c5] FINISH, SetVdsStatusVDSCommand, log id: 5144548a >2014-04-30 03:58:03,590 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [3cb2d7c5] Correlation ID: 3cb2d7c5, Job ID: eaf9a55b-e698-40ef-8e6d-0eaae9881256, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 03:58:03,597 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [3cb2d7c5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 03:58:03,599 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [3cb2d7c5] Correlation ID: 4ee48c09, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 03:58:03,603 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-32) [1f5eb940] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:58:03,604 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-32) [1edcaf5e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 03:58:03,605 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [1edcaf5e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-35) [3b5a09da] Autorecovering 1 hosts >2014-04-30 04:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-35) [3b5a09da] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7a3d9f91 >2014-04-30 04:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] FINISH, SetVdsStatusVDSCommand, log id: 7a3d9f91 >2014-04-30 04:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4d2d3006 >2014-04-30 04:00:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: b74ffb8 >2014-04-30 04:00:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] Failed in GetHardwareInfoVDS method >2014-04-30 04:00:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:00:00,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2e408ee8 >2014-04-30 04:00:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] HostName = 10.34.27.209 >2014-04-30 04:00:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:00:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-35) [4a1c0087] FINISH, GetHardwareInfoVDSCommand, log id: b74ffb8 >2014-04-30 04:00:00,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-35) [4a1c0087] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:00:00,730 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-35) [5da3a5ba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:00:00,734 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-35) [41281eb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:00:00,735 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-35) [41281eb] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4d2d3006 >2014-04-30 04:00:00,758 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [41281eb] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:00:00,759 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-35) [41281eb] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:00:03,956 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 72fa8196 >2014-04-30 04:00:04,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] Failed in GetHardwareInfoVDS method >2014-04-30 04:00:04,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:00:04,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1c51d9d3 >2014-04-30 04:00:04,178 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] HostName = 10.34.27.209 >2014-04-30 04:00:04,178 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:00:04,179 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-18) [3d2458a0] FINISH, GetHardwareInfoVDSCommand, log id: 72fa8196 >2014-04-30 04:00:04,183 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [3d2458a0] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:00:04,193 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [3d2458a0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:00:04,415 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-18) [390ae59] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:00:04,417 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-18) [40b068e5] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:00:04,423 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [40b068e5] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 18de0699 >2014-04-30 04:03:04,425 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [40b068e5] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:03:04,425 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-18) [40b068e5] FINISH, ConnectStorageServerVDSCommand, log id: 18de0699 >2014-04-30 04:03:04,425 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-18) [40b068e5] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:03:04,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 59eb18c1 >2014-04-30 04:03:04,869 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 59eb18c1 >2014-04-30 04:03:04,869 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:03:04,879 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-18) [1ec93f4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:03:04,880 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [1ec93f4e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 49b973a5 >2014-04-30 04:03:04,883 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-18) [1ec93f4e] FINISH, SetVdsStatusVDSCommand, log id: 49b973a5 >2014-04-30 04:03:04,889 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [1ec93f4e] Correlation ID: 1ec93f4e, Job ID: 6dcf349c-5cc5-4b53-8b3d-5274f7c58277, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:03:04,895 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [1ec93f4e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:03:04,900 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-18) [1ec93f4e] Correlation ID: 390ae59, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:03:04,904 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-18) [6833a5cc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:03:04,905 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-18) [25458d88] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:03:04,906 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-18) [25458d88] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:05:00,013 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) [4c9c7400] Autorecovering 1 hosts >2014-04-30 04:05:00,013 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-38) [4c9c7400] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:05:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:05:00,015 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:05:00,019 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:05:00,019 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:05:00,027 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 35832701 >2014-04-30 04:05:00,035 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] FINISH, SetVdsStatusVDSCommand, log id: 35832701 >2014-04-30 04:05:00,037 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4ab73f66 >2014-04-30 04:05:00,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 16154f71 >2014-04-30 04:05:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] Failed in GetHardwareInfoVDS method >2014-04-30 04:05:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:05:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@51128f0a >2014-04-30 04:05:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] HostName = 10.34.27.209 >2014-04-30 04:05:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:05:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-38) [3d5e672d] FINISH, GetHardwareInfoVDSCommand, log id: 16154f71 >2014-04-30 04:05:00,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-38) [3d5e672d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:05:00,769 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-38) [36605b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:05:00,770 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-38) [1e3e8109] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:05:00,771 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-38) [1e3e8109] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4ab73f66 >2014-04-30 04:05:00,786 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-38) [1e3e8109] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:05:00,788 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-38) [1e3e8109] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:05:02,271 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3a93831a >2014-04-30 04:05:02,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] Failed in GetHardwareInfoVDS method >2014-04-30 04:05:02,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:05:02,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2a843e01 >2014-04-30 04:05:02,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] HostName = 10.34.27.209 >2014-04-30 04:05:02,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:05:02,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4d552b21] FINISH, GetHardwareInfoVDSCommand, log id: 3a93831a >2014-04-30 04:05:02,498 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [4d552b21] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:05:02,507 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [4d552b21] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:05:02,730 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-4) [31941961] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:05:02,731 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-4) [42435025] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:05:02,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [42435025] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 44232c1c >2014-04-30 04:08:02,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [42435025] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:08:02,740 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [42435025] FINISH, ConnectStorageServerVDSCommand, log id: 44232c1c >2014-04-30 04:08:02,740 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-4) [42435025] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:08:02,743 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 462d197f >2014-04-30 04:08:03,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 462d197f >2014-04-30 04:08:03,196 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:08:03,218 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-4) [d92d4d5] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:08:03,219 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [d92d4d5] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4e13cd22 >2014-04-30 04:08:03,222 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [d92d4d5] FINISH, SetVdsStatusVDSCommand, log id: 4e13cd22 >2014-04-30 04:08:03,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [d92d4d5] Correlation ID: d92d4d5, Job ID: 5d8ea237-01df-4b2f-9246-1f5d7453e1f1, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:08:03,235 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [d92d4d5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:08:03,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [d92d4d5] Correlation ID: 31941961, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:08:03,242 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-4) [a613468] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:08:03,243 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-4) [64b2333f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:08:03,244 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) [64b2333f] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-17) [30691e32] Autorecovering 1 hosts >2014-04-30 04:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-17) [30691e32] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:10:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7a86d764 >2014-04-30 04:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] FINISH, SetVdsStatusVDSCommand, log id: 7a86d764 >2014-04-30 04:10:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6082b635 >2014-04-30 04:10:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6d405cc2 >2014-04-30 04:10:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] Failed in GetHardwareInfoVDS method >2014-04-30 04:10:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:10:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1c89fc97 >2014-04-30 04:10:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] HostName = 10.34.27.209 >2014-04-30 04:10:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:10:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [6f52774b] FINISH, GetHardwareInfoVDSCommand, log id: 6d405cc2 >2014-04-30 04:10:00,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [6f52774b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:10:00,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-17) [748dada5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:10:00,730 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-17) [2080472c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:10:00,731 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) [2080472c] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6082b635 >2014-04-30 04:10:00,744 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [2080472c] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:10:00,745 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [2080472c] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:10:03,611 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5913e0dd >2014-04-30 04:10:03,834 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] Failed in GetHardwareInfoVDS method >2014-04-30 04:10:03,834 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:10:03,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@280ffadb >2014-04-30 04:10:03,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] HostName = 10.34.27.209 >2014-04-30 04:10:03,834 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:10:03,834 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-48) [1e4d6827] FINISH, GetHardwareInfoVDSCommand, log id: 5913e0dd >2014-04-30 04:10:03,839 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [1e4d6827] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:10:03,848 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [1e4d6827] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:10:04,071 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-48) [4bb6102c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:10:04,073 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [3b01df7c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:10:04,079 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [3b01df7c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4da58273 >2014-04-30 04:13:04,080 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [3b01df7c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:13:04,080 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-48) [3b01df7c] FINISH, ConnectStorageServerVDSCommand, log id: 4da58273 >2014-04-30 04:13:04,080 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-48) [3b01df7c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:13:04,085 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 13315635 >2014-04-30 04:13:04,543 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 13315635 >2014-04-30 04:13:04,543 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:13:04,554 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-48) [6c6cd7dc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:13:04,554 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [6c6cd7dc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6d1917c9 >2014-04-30 04:13:04,557 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-48) [6c6cd7dc] FINISH, SetVdsStatusVDSCommand, log id: 6d1917c9 >2014-04-30 04:13:04,563 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [6c6cd7dc] Correlation ID: 6c6cd7dc, Job ID: 07a01216-931f-48c8-9eef-8b32579f3a8e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:13:04,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [6c6cd7dc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:13:04,573 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [6c6cd7dc] Correlation ID: 4bb6102c, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:13:04,577 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-48) [2d5cfbdc] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:13:04,578 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-48) [4c69ee03] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:13:04,578 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-48) [4c69ee03] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-64) [303a359a] Autorecovering 1 hosts >2014-04-30 04:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-64) [303a359a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:15:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 57076998 >2014-04-30 04:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] FINISH, SetVdsStatusVDSCommand, log id: 57076998 >2014-04-30 04:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 69998b4f >2014-04-30 04:15:00,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 50f6e9ab >2014-04-30 04:15:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] Failed in GetHardwareInfoVDS method >2014-04-30 04:15:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:15:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2670314 >2014-04-30 04:15:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] HostName = 10.34.27.209 >2014-04-30 04:15:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:15:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [491323e9] FINISH, GetHardwareInfoVDSCommand, log id: 50f6e9ab >2014-04-30 04:15:00,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [491323e9] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:15:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [12cb262d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:15:00,716 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:15:00,717 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 69998b4f >2014-04-30 04:15:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:15:00,732 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:15:01,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [5ee91683] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 738c993c >2014-04-30 04:15:02,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [5ee91683] Failed in GetHardwareInfoVDS method >2014-04-30 04:15:02,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [5ee91683] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:15:02,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [5ee91683] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@96ed0e9 >2014-04-30 04:15:02,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [5ee91683] HostName = 10.34.27.209 >2014-04-30 04:15:02,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [5ee91683] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:15:02,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [5ee91683] FINISH, GetHardwareInfoVDSCommand, log id: 738c993c >2014-04-30 04:15:02,165 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [5ee91683] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:15:02,177 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [5ee91683] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:15:02,400 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-27) [551faded] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:15:02,402 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-27) [5cd6645a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:15:02,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-27) [5cd6645a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3c7bde2d >2014-04-30 04:18:02,409 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-27) [5cd6645a] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:18:02,409 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-27) [5cd6645a] FINISH, ConnectStorageServerVDSCommand, log id: 3c7bde2d >2014-04-30 04:18:02,409 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-27) [5cd6645a] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:18:02,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 18e7febd >2014-04-30 04:18:02,875 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 18e7febd >2014-04-30 04:18:02,875 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:18:02,885 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-27) [6f0e00db] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:18:02,886 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) [6f0e00db] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4efaf126 >2014-04-30 04:18:02,889 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) [6f0e00db] FINISH, SetVdsStatusVDSCommand, log id: 4efaf126 >2014-04-30 04:18:02,894 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [6f0e00db] Correlation ID: 6f0e00db, Job ID: 1be5734f-e522-4c3f-89df-c002a25b2dda, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:18:02,900 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [6f0e00db] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:18:02,903 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [6f0e00db] Correlation ID: 551faded, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:18:02,906 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-27) [28122da6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:18:02,907 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-27) [61de2f6e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:18:02,908 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) [61de2f6e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [46147fed] Autorecovering 1 hosts >2014-04-30 04:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-56) [46147fed] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4f3f0dbc >2014-04-30 04:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] FINISH, SetVdsStatusVDSCommand, log id: 4f3f0dbc >2014-04-30 04:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4cb03bc4 >2014-04-30 04:20:00,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 123c2bed >2014-04-30 04:20:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] Failed in GetHardwareInfoVDS method >2014-04-30 04:20:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:20:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2c08c5dd >2014-04-30 04:20:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] HostName = 10.34.27.209 >2014-04-30 04:20:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:20:00,711 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] FINISH, GetHardwareInfoVDSCommand, log id: 123c2bed >2014-04-30 04:20:00,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-56) [37d0b4c9] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:20:00,725 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-56) [104b75bd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:20:00,727 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-56) [4e006a9c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:20:00,727 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-56) [4e006a9c] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4cb03bc4 >2014-04-30 04:20:00,741 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [4e006a9c] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:20:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-56) [4e006a9c] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:20:03,268 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 13127764 >2014-04-30 04:20:03,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] Failed in GetHardwareInfoVDS method >2014-04-30 04:20:03,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:20:03,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@215c5004 >2014-04-30 04:20:03,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] HostName = 10.34.27.209 >2014-04-30 04:20:03,492 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:20:03,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [458d0cd3] FINISH, GetHardwareInfoVDSCommand, log id: 13127764 >2014-04-30 04:20:03,497 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [458d0cd3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:20:03,506 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [458d0cd3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:20:03,728 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-31) [59c82b5b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:20:03,729 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [47f156d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:20:03,737 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [47f156d3] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 605cf7d1 >2014-04-30 04:23:03,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [47f156d3] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:23:03,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-31) [47f156d3] FINISH, ConnectStorageServerVDSCommand, log id: 605cf7d1 >2014-04-30 04:23:03,739 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-31) [47f156d3] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:23:03,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 28a61810 >2014-04-30 04:23:04,207 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 28a61810 >2014-04-30 04:23:04,207 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:23:04,239 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-31) [2a557442] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:23:04,240 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [2a557442] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 17907a43 >2014-04-30 04:23:04,249 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [2a557442] FINISH, SetVdsStatusVDSCommand, log id: 17907a43 >2014-04-30 04:23:04,255 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [2a557442] Correlation ID: 2a557442, Job ID: d0fc812e-f94c-4c7b-9cd2-b29c3ea687c6, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:23:04,265 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [2a557442] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:23:04,268 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [2a557442] Correlation ID: 59c82b5b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:23:04,272 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-31) [dfa4777] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:23:04,273 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-31) [9f57b25] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:23:04,274 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-31) [9f57b25] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-6) [4e1b8719] Autorecovering 1 hosts >2014-04-30 04:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-6) [4e1b8719] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 49f37fd5 >2014-04-30 04:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] FINISH, SetVdsStatusVDSCommand, log id: 49f37fd5 >2014-04-30 04:25:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 38cfea61 >2014-04-30 04:25:00,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 45466f7b >2014-04-30 04:25:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] Failed in GetHardwareInfoVDS method >2014-04-30 04:25:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:25:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5fd5c180 >2014-04-30 04:25:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] HostName = 10.34.27.209 >2014-04-30 04:25:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:25:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [70b6366d] FINISH, GetHardwareInfoVDSCommand, log id: 45466f7b >2014-04-30 04:25:00,717 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [70b6366d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:25:00,725 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [6c68618f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:25:00,726 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [3f865792] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:25:00,726 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-6) [3f865792] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 38cfea61 >2014-04-30 04:25:00,743 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [3f865792] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:25:00,744 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [3f865792] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:25:01,625 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 60dabfb >2014-04-30 04:25:01,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] Failed in GetHardwareInfoVDS method >2014-04-30 04:25:01,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:25:01,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6b95eed7 >2014-04-30 04:25:01,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] HostName = 10.34.27.209 >2014-04-30 04:25:01,848 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:25:01,848 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [2f9c813e] FINISH, GetHardwareInfoVDSCommand, log id: 60dabfb >2014-04-30 04:25:01,854 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [2f9c813e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:25:01,863 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [2f9c813e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:25:02,084 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-50) [2e0e3bef] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:25:02,086 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [916ce6b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:25:02,093 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [916ce6b] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 19209b9d >2014-04-30 04:28:02,095 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [916ce6b] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:28:02,095 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [916ce6b] FINISH, ConnectStorageServerVDSCommand, log id: 19209b9d >2014-04-30 04:28:02,095 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [916ce6b] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:28:02,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5095018 >2014-04-30 04:28:02,565 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 5095018 >2014-04-30 04:28:02,565 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:28:02,577 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-50) [568f9be0] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:28:02,578 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [568f9be0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 44df43e4 >2014-04-30 04:28:02,581 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [568f9be0] FINISH, SetVdsStatusVDSCommand, log id: 44df43e4 >2014-04-30 04:28:02,587 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [568f9be0] Correlation ID: 568f9be0, Job ID: 7b770084-4797-4e21-b9ae-40e7d865d798, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:28:02,595 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [568f9be0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:28:02,597 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [568f9be0] Correlation ID: 2e0e3bef, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:28:02,602 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [756c8e30] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:28:02,603 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [7844daf7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:28:02,604 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [7844daf7] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:30:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-16) [53de3c6d] Autorecovering 1 hosts >2014-04-30 04:30:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-16) [53de3c6d] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [71105816] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [71105816] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [71105816] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:30:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [71105816] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [71105816] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 65f6fb36 >2014-04-30 04:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [71105816] FINISH, SetVdsStatusVDSCommand, log id: 65f6fb36 >2014-04-30 04:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) [71105816] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6f1caca9 >2014-04-30 04:30:00,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [71105816] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3ebfc45f >2014-04-30 04:30:00,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [71105816] Failed in GetHardwareInfoVDS method >2014-04-30 04:30:00,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [71105816] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:30:00,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [71105816] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@665ee0ef >2014-04-30 04:30:00,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [71105816] HostName = 10.34.27.209 >2014-04-30 04:30:00,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [71105816] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:30:00,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [71105816] FINISH, GetHardwareInfoVDSCommand, log id: 3ebfc45f >2014-04-30 04:30:00,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [71105816] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:30:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-16) [12a3ffc3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:30:00,732 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:30:00,733 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6f1caca9 >2014-04-30 04:30:00,750 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:30:00,751 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:30:02,964 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [7327d517] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 45b79109 >2014-04-30 04:30:03,186 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [7327d517] Failed in GetHardwareInfoVDS method >2014-04-30 04:30:03,186 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [7327d517] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:30:03,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [7327d517] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@169309a1 >2014-04-30 04:30:03,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [7327d517] HostName = 10.34.27.209 >2014-04-30 04:30:03,186 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [7327d517] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:30:03,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [7327d517] FINISH, GetHardwareInfoVDSCommand, log id: 45b79109 >2014-04-30 04:30:03,192 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [7327d517] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:30:03,216 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [7327d517] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:30:03,439 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [756fde10] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:30:03,440 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [5f669555] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:30:03,448 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f669555] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 52cfdc0a >2014-04-30 04:33:03,454 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f669555] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:33:03,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [5f669555] FINISH, ConnectStorageServerVDSCommand, log id: 52cfdc0a >2014-04-30 04:33:03,454 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [5f669555] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:33:03,458 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5145d0c5 >2014-04-30 04:33:03,899 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 5145d0c5 >2014-04-30 04:33:03,899 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:33:03,909 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-23) [2ad56e20] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:33:03,910 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [2ad56e20] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2f562f72 >2014-04-30 04:33:03,912 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [2ad56e20] FINISH, SetVdsStatusVDSCommand, log id: 2f562f72 >2014-04-30 04:33:03,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2ad56e20] Correlation ID: 2ad56e20, Job ID: 0d9e69ad-3ff5-4931-9002-d3e55d8a2820, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:33:03,924 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2ad56e20] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:33:03,926 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [2ad56e20] Correlation ID: 756fde10, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:33:03,929 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-23) [2aa1009f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:33:03,930 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-23) [1469ce16] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:33:03,931 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) [1469ce16] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [1b5b347b] Autorecovering 1 hosts >2014-04-30 04:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-77) [1b5b347b] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:35:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:35:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:35:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:35:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:35:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 24f6190c >2014-04-30 04:35:00,019 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] FINISH, SetVdsStatusVDSCommand, log id: 24f6190c >2014-04-30 04:35:00,021 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 38a5e073 >2014-04-30 04:35:00,487 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 375de650 >2014-04-30 04:35:00,722 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] Failed in GetHardwareInfoVDS method >2014-04-30 04:35:00,722 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:35:00,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2f2581c0 >2014-04-30 04:35:00,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] HostName = 10.34.27.209 >2014-04-30 04:35:00,722 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:35:00,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-77) [76abdd01] FINISH, GetHardwareInfoVDSCommand, log id: 375de650 >2014-04-30 04:35:00,728 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-77) [76abdd01] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:35:00,738 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-77) [5f0cb19d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:35:00,739 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-77) [201b139] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:35:00,740 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-77) [201b139] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 38a5e073 >2014-04-30 04:35:00,754 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [201b139] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:35:00,755 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-77) [201b139] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:35:01,290 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3611d5a7 >2014-04-30 04:35:01,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] Failed in GetHardwareInfoVDS method >2014-04-30 04:35:01,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:35:01,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4ce1e2ac >2014-04-30 04:35:01,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] HostName = 10.34.27.209 >2014-04-30 04:35:01,513 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:35:01,513 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [57f94eec] FINISH, GetHardwareInfoVDSCommand, log id: 3611d5a7 >2014-04-30 04:35:01,519 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [57f94eec] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:35:01,534 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [57f94eec] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:35:01,755 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-45) [63dfea12] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:35:01,756 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [7bd0505c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:35:01,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [7bd0505c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3193ed72 >2014-04-30 04:36:05,022 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-98) [48e5821f] Start refreshing all users data >2014-04-30 04:36:07,303 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [ca31efd] Attempting to update VMs/Templates Ovf. >2014-04-30 04:36:07,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [ca31efd] Attempting to update VM OVFs in Data Center NFS >2014-04-30 04:36:07,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [ca31efd] Successfully updated VM OVFs in Data Center NFS >2014-04-30 04:36:07,305 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [ca31efd] Attempting to update template OVFs in Data Center NFS >2014-04-30 04:36:07,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [ca31efd] Successfully updated templates OVFs in Data Center NFS >2014-04-30 04:36:07,306 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [ca31efd] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-30 04:36:07,307 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-21) [ca31efd] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-30 04:38:01,764 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [7bd0505c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:38:01,764 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [7bd0505c] FINISH, ConnectStorageServerVDSCommand, log id: 3193ed72 >2014-04-30 04:38:01,764 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [7bd0505c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:38:01,768 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 416e5d68 >2014-04-30 04:38:02,208 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 416e5d68 >2014-04-30 04:38:02,208 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:38:02,220 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-45) [2106c1ab] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:38:02,221 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [2106c1ab] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 31957db3 >2014-04-30 04:38:02,223 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [2106c1ab] FINISH, SetVdsStatusVDSCommand, log id: 31957db3 >2014-04-30 04:38:02,229 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [2106c1ab] Correlation ID: 2106c1ab, Job ID: 6326fefe-9529-4803-80e6-3d0502b4ac12, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:38:02,235 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [2106c1ab] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:38:02,237 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [2106c1ab] Correlation ID: 63dfea12, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:38:02,243 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [629161b5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:38:02,244 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [409cceea] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:38:02,244 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) [409cceea] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-76) [3bd66c3a] Autorecovering 1 hosts >2014-04-30 04:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-76) [3bd66c3a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 536b86fb >2014-04-30 04:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] FINISH, SetVdsStatusVDSCommand, log id: 536b86fb >2014-04-30 04:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3938618f >2014-04-30 04:40:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: ac43d75 >2014-04-30 04:40:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] Failed in GetHardwareInfoVDS method >2014-04-30 04:40:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:40:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@48957c5c >2014-04-30 04:40:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] HostName = 10.34.27.209 >2014-04-30 04:40:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:40:00,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [54ba052e] FINISH, GetHardwareInfoVDSCommand, log id: ac43d75 >2014-04-30 04:40:00,709 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [54ba052e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:40:00,752 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-76) [459a4646] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:40:00,753 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:40:00,754 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3938618f >2014-04-30 04:40:00,769 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:40:00,770 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:40:02,608 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5805681a >2014-04-30 04:40:02,830 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] Failed in GetHardwareInfoVDS method >2014-04-30 04:40:02,830 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:40:02,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4af5b01b >2014-04-30 04:40:02,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] HostName = 10.34.27.209 >2014-04-30 04:40:02,830 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:40:02,830 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [67b7f665] FINISH, GetHardwareInfoVDSCommand, log id: 5805681a >2014-04-30 04:40:02,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [67b7f665] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:40:02,844 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [67b7f665] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:40:03,069 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-55) [191f80b3] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:40:03,072 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [62a57254] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:40:03,082 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [62a57254] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6790ead9 >2014-04-30 04:43:03,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [62a57254] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:43:03,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [62a57254] FINISH, ConnectStorageServerVDSCommand, log id: 6790ead9 >2014-04-30 04:43:03,084 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [62a57254] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:43:03,088 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 234b2cb2 >2014-04-30 04:43:03,527 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 234b2cb2 >2014-04-30 04:43:03,527 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:43:03,547 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-55) [4e77a91c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:43:03,549 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [4e77a91c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: a13d29 >2014-04-30 04:43:03,552 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [4e77a91c] FINISH, SetVdsStatusVDSCommand, log id: a13d29 >2014-04-30 04:43:03,559 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [4e77a91c] Correlation ID: 4e77a91c, Job ID: ae38f895-5d7e-42c4-8570-ba9847c962d2, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:43:03,565 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [4e77a91c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:43:03,568 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [4e77a91c] Correlation ID: 191f80b3, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:43:03,573 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [69aa7119] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:43:03,574 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [24676a68] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:43:03,575 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [24676a68] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-41) [2fc678f0] Autorecovering 1 hosts >2014-04-30 04:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-41) [2fc678f0] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:45:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:45:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4b4444da >2014-04-30 04:45:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] FINISH, SetVdsStatusVDSCommand, log id: 4b4444da >2014-04-30 04:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6c9e0794 >2014-04-30 04:45:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 446102c3 >2014-04-30 04:45:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] Failed in GetHardwareInfoVDS method >2014-04-30 04:45:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:45:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6a34213a >2014-04-30 04:45:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] HostName = 10.34.27.209 >2014-04-30 04:45:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:45:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-41) [7b0424ec] FINISH, GetHardwareInfoVDSCommand, log id: 446102c3 >2014-04-30 04:45:00,710 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-41) [7b0424ec] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:45:00,723 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-41) [1f2d047c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:45:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-41) [b910aff] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:45:00,725 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-41) [b910aff] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6c9e0794 >2014-04-30 04:45:00,738 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [b910aff] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:45:00,739 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [b910aff] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:45:03,936 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 62d7c2d1 >2014-04-30 04:45:04,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] Failed in GetHardwareInfoVDS method >2014-04-30 04:45:04,158 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:45:04,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@43375a09 >2014-04-30 04:45:04,158 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] HostName = 10.34.27.209 >2014-04-30 04:45:04,159 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:45:04,159 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] FINISH, GetHardwareInfoVDSCommand, log id: 62d7c2d1 >2014-04-30 04:45:04,371 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:45:04,398 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [6e8d9be5] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:45:04,620 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-13) [55d3f87f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:45:04,622 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-13) [5734c6dd] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:45:04,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [5734c6dd] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1213e8 >2014-04-30 04:48:04,633 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [5734c6dd] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:48:04,633 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [5734c6dd] FINISH, ConnectStorageServerVDSCommand, log id: 1213e8 >2014-04-30 04:48:04,633 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-13) [5734c6dd] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:48:04,637 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 30f4fc2 >2014-04-30 04:48:05,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 30f4fc2 >2014-04-30 04:48:05,078 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:48:05,134 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-13) [11d171e9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:48:05,135 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [11d171e9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1482ee81 >2014-04-30 04:48:05,138 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [11d171e9] FINISH, SetVdsStatusVDSCommand, log id: 1482ee81 >2014-04-30 04:48:05,144 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [11d171e9] Correlation ID: 11d171e9, Job ID: a1064a17-bcce-4c89-82a3-4f163091a0d6, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:48:05,152 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [11d171e9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:48:05,154 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [11d171e9] Correlation ID: 55d3f87f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:48:05,157 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-13) [4b4a6ba] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:48:05,158 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-13) [451e95f6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:48:05,159 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [451e95f6] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:48:25,375 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-48) [4c69ee03] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Warning, Low disk space.storage domain has 3 GB of free space >2014-04-30 04:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-82) [46150d65] Autorecovering 1 hosts >2014-04-30 04:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-82) [46150d65] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [72758355] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [72758355] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [72758355] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [72758355] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [72758355] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2480b3f4 >2014-04-30 04:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [72758355] FINISH, SetVdsStatusVDSCommand, log id: 2480b3f4 >2014-04-30 04:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) [72758355] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 53f45dd1 >2014-04-30 04:50:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [72758355] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: c10e5bb >2014-04-30 04:50:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [72758355] Failed in GetHardwareInfoVDS method >2014-04-30 04:50:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [72758355] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:50:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [72758355] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@550f2e4c >2014-04-30 04:50:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [72758355] HostName = 10.34.27.209 >2014-04-30 04:50:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [72758355] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:50:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [72758355] FINISH, GetHardwareInfoVDSCommand, log id: c10e5bb >2014-04-30 04:50:00,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [72758355] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:50:00,712 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-82) [61b397c1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:50:00,713 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-82) [24d883ab] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:50:00,714 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) [24d883ab] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 53f45dd1 >2014-04-30 04:50:00,727 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [24d883ab] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:50:00,728 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [24d883ab] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:50:02,512 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [7844daf7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3dafc5ea >2014-04-30 04:50:02,734 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [7844daf7] Failed in GetHardwareInfoVDS method >2014-04-30 04:50:02,734 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [7844daf7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:50:02,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [7844daf7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5eeaa549 >2014-04-30 04:50:02,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [7844daf7] HostName = 10.34.27.209 >2014-04-30 04:50:02,734 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [7844daf7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:50:02,734 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-50) [7844daf7] FINISH, GetHardwareInfoVDSCommand, log id: 3dafc5ea >2014-04-30 04:50:02,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [7844daf7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:50:02,748 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [7844daf7] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:50:02,986 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-50) [379e506f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:50:02,988 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [6cfa45ad] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:50:02,994 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [6cfa45ad] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 240dfa23 >2014-04-30 04:53:03,001 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [6cfa45ad] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:53:03,002 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-50) [6cfa45ad] FINISH, ConnectStorageServerVDSCommand, log id: 240dfa23 >2014-04-30 04:53:03,002 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-50) [6cfa45ad] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:53:03,006 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5c5c57e6 >2014-04-30 04:53:03,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5c5c57e6 >2014-04-30 04:53:03,454 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:53:03,467 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-50) [673faeeb] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:53:03,469 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [673faeeb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2057853e >2014-04-30 04:53:03,472 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-50) [673faeeb] FINISH, SetVdsStatusVDSCommand, log id: 2057853e >2014-04-30 04:53:03,480 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [673faeeb] Correlation ID: 673faeeb, Job ID: 82ebadc4-dfc4-45bc-ab05-392ba1d99bce, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:53:03,489 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [673faeeb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:53:03,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-50) [673faeeb] Correlation ID: 379e506f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:53:03,498 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-50) [c7bdfdd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:53:03,499 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-50) [39cbe00c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:53:03,500 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-50) [39cbe00c] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 04:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) [1355ae52] Autorecovering 1 hosts >2014-04-30 04:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-78) [1355ae52] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 04:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 04:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 04:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: b87fad4 >2014-04-30 04:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] FINISH, SetVdsStatusVDSCommand, log id: b87fad4 >2014-04-30 04:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5e68f391 >2014-04-30 04:55:00,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 32998dec >2014-04-30 04:55:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] Failed in GetHardwareInfoVDS method >2014-04-30 04:55:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:55:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@58e23452 >2014-04-30 04:55:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] HostName = 10.34.27.209 >2014-04-30 04:55:00,718 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:55:00,718 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] FINISH, GetHardwareInfoVDSCommand, log id: 32998dec >2014-04-30 04:55:00,723 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [4f0f63cb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:55:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [6736e331] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:55:00,732 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:55:00,733 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5e68f391 >2014-04-30 04:55:00,746 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 04:55:00,747 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 04:55:03,854 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [1469ce16] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2ab425ae >2014-04-30 04:55:04,077 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [1469ce16] Failed in GetHardwareInfoVDS method >2014-04-30 04:55:04,078 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [1469ce16] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:55:04,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [1469ce16] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1915d869 >2014-04-30 04:55:04,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [1469ce16] HostName = 10.34.27.209 >2014-04-30 04:55:04,078 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [1469ce16] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 04:55:04,078 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-23) [1469ce16] FINISH, GetHardwareInfoVDSCommand, log id: 2ab425ae >2014-04-30 04:55:04,084 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1469ce16] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 04:55:04,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [1469ce16] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 04:55:04,321 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-23) [63c07342] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:55:04,323 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [3b4b35ed] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 04:55:04,333 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [3b4b35ed] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 70b6d2c6 >2014-04-30 04:58:04,336 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [3b4b35ed] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 04:58:04,336 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-23) [3b4b35ed] FINISH, ConnectStorageServerVDSCommand, log id: 70b6d2c6 >2014-04-30 04:58:04,336 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-23) [3b4b35ed] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 04:58:04,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 535efadb >2014-04-30 04:58:04,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 535efadb >2014-04-30 04:58:04,794 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 04:58:04,805 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-23) [426b61d4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:58:04,806 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [426b61d4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2f6da0b9 >2014-04-30 04:58:04,809 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-23) [426b61d4] FINISH, SetVdsStatusVDSCommand, log id: 2f6da0b9 >2014-04-30 04:58:04,815 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [426b61d4] Correlation ID: 426b61d4, Job ID: 8e3e95d7-8917-484b-a410-8891ab2d40e3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 04:58:04,824 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [426b61d4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 04:58:04,826 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-23) [426b61d4] Correlation ID: 63c07342, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 04:58:04,830 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-23) [1fe17a5b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:58:04,832 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-23) [7bb09a3d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 04:58:04,832 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-23) [7bb09a3d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-71) [635c6af7] Autorecovering 1 hosts >2014-04-30 05:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-71) [635c6af7] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:00:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 54e794d1 >2014-04-30 05:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] FINISH, SetVdsStatusVDSCommand, log id: 54e794d1 >2014-04-30 05:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 743b1ec8 >2014-04-30 05:00:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 37242d42 >2014-04-30 05:00:00,696 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] Failed in GetHardwareInfoVDS method >2014-04-30 05:00:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:00:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@28176456 >2014-04-30 05:00:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] HostName = 10.34.27.209 >2014-04-30 05:00:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:00:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [51131c1b] FINISH, GetHardwareInfoVDSCommand, log id: 37242d42 >2014-04-30 05:00:00,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [51131c1b] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:00:00,710 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-71) [34d16e4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:00:00,711 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:00:00,712 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 743b1ec8 >2014-04-30 05:00:00,725 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:00:00,726 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:00:02,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 72b9f9ce >2014-04-30 05:00:02,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] Failed in GetHardwareInfoVDS method >2014-04-30 05:00:02,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:00:02,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@fbf7e3a >2014-04-30 05:00:02,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] HostName = 10.34.27.209 >2014-04-30 05:00:02,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:00:02,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] FINISH, GetHardwareInfoVDSCommand, log id: 72b9f9ce >2014-04-30 05:00:02,414 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:00:02,423 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [3ad94d5e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:00:02,644 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-26) [7523d9e2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:00:02,646 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [4c07f706] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:00:02,653 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [4c07f706] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4f2a3938 >2014-04-30 05:03:02,655 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [4c07f706] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:03:02,655 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [4c07f706] FINISH, ConnectStorageServerVDSCommand, log id: 4f2a3938 >2014-04-30 05:03:02,655 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [4c07f706] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:03:02,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4451fb00 >2014-04-30 05:03:03,099 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4451fb00 >2014-04-30 05:03:03,099 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:03:03,111 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-26) [798e1710] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:03:03,112 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [798e1710] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6098ccec >2014-04-30 05:03:03,115 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [798e1710] FINISH, SetVdsStatusVDSCommand, log id: 6098ccec >2014-04-30 05:03:03,123 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [798e1710] Correlation ID: 798e1710, Job ID: c1eaf18e-1575-48d4-91a4-f285f0a87be8, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:03:03,129 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [798e1710] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:03:03,131 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [798e1710] Correlation ID: 7523d9e2, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:03:03,135 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-26) [410848a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:03:03,137 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-26) [3b213f62] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:03:03,138 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [3b213f62] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [4e9fed15] Autorecovering 1 hosts >2014-04-30 05:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [4e9fed15] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 69d8129b >2014-04-30 05:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] FINISH, SetVdsStatusVDSCommand, log id: 69d8129b >2014-04-30 05:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 11828652 >2014-04-30 05:05:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 29037f6a >2014-04-30 05:05:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] Failed in GetHardwareInfoVDS method >2014-04-30 05:05:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:05:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@24a21d2f >2014-04-30 05:05:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] HostName = 10.34.27.209 >2014-04-30 05:05:00,707 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:05:00,707 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [3127f9ca] FINISH, GetHardwareInfoVDSCommand, log id: 29037f6a >2014-04-30 05:05:00,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [3127f9ca] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:05:00,730 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [7480d43b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:05:00,732 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [483bbf17] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:05:00,732 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [483bbf17] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 11828652 >2014-04-30 05:05:00,746 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [483bbf17] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:05:00,747 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [483bbf17] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:05:03,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [403277f3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4b99c80c >2014-04-30 05:05:03,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [403277f3] Failed in GetHardwareInfoVDS method >2014-04-30 05:05:03,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [403277f3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:05:03,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [403277f3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4a81efe0 >2014-04-30 05:05:03,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [403277f3] HostName = 10.34.27.209 >2014-04-30 05:05:03,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [403277f3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:05:03,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-15) [403277f3] FINISH, GetHardwareInfoVDSCommand, log id: 4b99c80c >2014-04-30 05:05:03,768 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [403277f3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:05:03,808 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [403277f3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:05:04,029 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-15) [38b5746] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:05:04,031 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-15) [2083268d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:05:04,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [2083268d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2fc46361 >2014-04-30 05:08:04,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [2083268d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:08:04,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-15) [2083268d] FINISH, ConnectStorageServerVDSCommand, log id: 2fc46361 >2014-04-30 05:08:04,038 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-15) [2083268d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:08:04,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 44dd20ea >2014-04-30 05:08:04,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 44dd20ea >2014-04-30 05:08:04,485 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:08:04,498 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-15) [12d9f88a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:08:04,499 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [12d9f88a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2743096 >2014-04-30 05:08:04,502 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-15) [12d9f88a] FINISH, SetVdsStatusVDSCommand, log id: 2743096 >2014-04-30 05:08:04,508 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [12d9f88a] Correlation ID: 12d9f88a, Job ID: 06ad15a2-37b5-484f-b700-1d1406a9bba0, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:08:04,515 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [12d9f88a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:08:04,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-15) [12d9f88a] Correlation ID: 38b5746, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:08:04,521 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-15) [3f62caf0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:08:04,522 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-15) [7af70e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:08:04,523 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-15) [7af70e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-31) [9f57b25] Autorecovering 1 hosts >2014-04-30 05:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-31) [9f57b25] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:10:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:10:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 433a7c65 >2014-04-30 05:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] FINISH, SetVdsStatusVDSCommand, log id: 433a7c65 >2014-04-30 05:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5138339 >2014-04-30 05:10:00,564 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4b581f7b >2014-04-30 05:10:00,801 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] Failed in GetHardwareInfoVDS method >2014-04-30 05:10:00,801 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:10:00,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@8b795bb >2014-04-30 05:10:00,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] HostName = 10.34.27.209 >2014-04-30 05:10:00,801 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:10:00,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [306b0510] FINISH, GetHardwareInfoVDSCommand, log id: 4b581f7b >2014-04-30 05:10:00,807 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [306b0510] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:10:00,818 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-31) [5886cebe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:10:00,820 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-31) [38bf1fe4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:10:00,820 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) [38bf1fe4] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5138339 >2014-04-30 05:10:00,840 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [38bf1fe4] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:10:00,842 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [38bf1fe4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:10:01,973 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: e8d2038 >2014-04-30 05:10:02,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] Failed in GetHardwareInfoVDS method >2014-04-30 05:10:02,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:10:02,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@533e43fe >2014-04-30 05:10:02,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] HostName = 10.34.27.209 >2014-04-30 05:10:02,195 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:10:02,195 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-42) [1770113a] FINISH, GetHardwareInfoVDSCommand, log id: e8d2038 >2014-04-30 05:10:02,200 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [1770113a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:10:02,209 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [1770113a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:10:02,433 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-42) [4227f2d8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:10:02,434 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-42) [4bbd7505] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:10:02,440 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-42) [4bbd7505] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4ecf7695 >2014-04-30 05:13:02,441 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-42) [4bbd7505] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:13:02,442 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-42) [4bbd7505] FINISH, ConnectStorageServerVDSCommand, log id: 4ecf7695 >2014-04-30 05:13:02,442 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-42) [4bbd7505] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:13:02,450 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3c665241 >2014-04-30 05:13:02,911 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 3c665241 >2014-04-30 05:13:02,912 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:13:02,923 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-42) [324d565d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:13:02,924 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [324d565d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2588f6ec >2014-04-30 05:13:02,926 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-42) [324d565d] FINISH, SetVdsStatusVDSCommand, log id: 2588f6ec >2014-04-30 05:13:02,932 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [324d565d] Correlation ID: 324d565d, Job ID: b7cdd0da-da89-4dd9-b094-cc912fd88c6f, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:13:02,939 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [324d565d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:13:02,941 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-42) [324d565d] Correlation ID: 4227f2d8, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:13:02,945 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-42) [6552a4b7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:13:02,946 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-42) [19235d27] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:13:02,947 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-42) [19235d27] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [5e757262] Autorecovering 1 hosts >2014-04-30 05:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [5e757262] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2274c1c1 >2014-04-30 05:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] FINISH, SetVdsStatusVDSCommand, log id: 2274c1c1 >2014-04-30 05:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 26fdfdef >2014-04-30 05:15:00,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 378656b8 >2014-04-30 05:15:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] Failed in GetHardwareInfoVDS method >2014-04-30 05:15:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:15:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@54a76682 >2014-04-30 05:15:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] HostName = 10.34.27.209 >2014-04-30 05:15:00,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:15:00,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [622c7b4a] FINISH, GetHardwareInfoVDSCommand, log id: 378656b8 >2014-04-30 05:15:00,708 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [622c7b4a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:15:00,718 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [64e1732b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:15:00,747 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [780ec1a0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:15:00,749 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [780ec1a0] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 26fdfdef >2014-04-30 05:15:00,766 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [780ec1a0] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:15:00,767 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [780ec1a0] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:15:03,315 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 579f6c85 >2014-04-30 05:15:03,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) Failed in GetHardwareInfoVDS method >2014-04-30 05:15:03,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:15:03,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@28fd8e4c >2014-04-30 05:15:03,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) HostName = 10.34.27.209 >2014-04-30 05:15:03,537 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:15:03,537 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) FINISH, GetHardwareInfoVDSCommand, log id: 579f6c85 >2014-04-30 05:15:03,541 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:15:03,551 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:15:03,774 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-39) [36888a0d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:15:03,776 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [429975a1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:15:03,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [429975a1] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5e62e7a4 >2014-04-30 05:18:03,784 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [429975a1] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:18:03,784 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-39) [429975a1] FINISH, ConnectStorageServerVDSCommand, log id: 5e62e7a4 >2014-04-30 05:18:03,784 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-39) [429975a1] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:18:03,787 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 67518475 >2014-04-30 05:18:04,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 67518475 >2014-04-30 05:18:04,390 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:18:04,401 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-39) [1db14a4e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:18:04,402 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-39) [1db14a4e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5eba93ed >2014-04-30 05:18:04,407 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-39) [1db14a4e] FINISH, SetVdsStatusVDSCommand, log id: 5eba93ed >2014-04-30 05:18:04,413 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [1db14a4e] Correlation ID: 1db14a4e, Job ID: 9f32c13f-6d7e-4907-b15b-f6f38363b927, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:18:04,420 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [1db14a4e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:18:04,422 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [1db14a4e] Correlation ID: 36888a0d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:18:04,427 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-39) [31f7dbe3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:18:04,428 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-39) [4d62b41d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:18:04,429 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-39) [4d62b41d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [1b2eb2bc] Autorecovering 1 hosts >2014-04-30 05:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [1b2eb2bc] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:20:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 40033e8d >2014-04-30 05:20:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] FINISH, SetVdsStatusVDSCommand, log id: 40033e8d >2014-04-30 05:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 271d0edf >2014-04-30 05:20:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7f3f43b8 >2014-04-30 05:20:00,696 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] Failed in GetHardwareInfoVDS method >2014-04-30 05:20:00,696 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:20:00,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1277264c >2014-04-30 05:20:00,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] HostName = 10.34.27.209 >2014-04-30 05:20:00,696 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:20:00,696 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [6b6838ec] FINISH, GetHardwareInfoVDSCommand, log id: 7f3f43b8 >2014-04-30 05:20:00,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [6b6838ec] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:20:00,712 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [a6d537c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:20:00,713 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [794dcbc1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:20:00,714 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [794dcbc1] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 271d0edf >2014-04-30 05:20:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [794dcbc1] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:20:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [794dcbc1] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:20:01,779 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4eb2e966] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 505c16d7 >2014-04-30 05:20:02,002 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4eb2e966] Failed in GetHardwareInfoVDS method >2014-04-30 05:20:02,003 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4eb2e966] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:20:02,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4eb2e966] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@48fbfac0 >2014-04-30 05:20:02,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4eb2e966] HostName = 10.34.27.209 >2014-04-30 05:20:02,003 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4eb2e966] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:20:02,003 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-94) [4eb2e966] FINISH, GetHardwareInfoVDSCommand, log id: 505c16d7 >2014-04-30 05:20:02,008 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [4eb2e966] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:20:02,018 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [4eb2e966] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:20:02,240 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-94) [7fb77095] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:20:02,242 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-94) [3a24c8b0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:20:02,248 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-94) [3a24c8b0] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 274bd099 >2014-04-30 05:23:02,250 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-94) [3a24c8b0] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:23:02,250 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-94) [3a24c8b0] FINISH, ConnectStorageServerVDSCommand, log id: 274bd099 >2014-04-30 05:23:02,251 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-94) [3a24c8b0] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:23:02,255 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 34a96274 >2014-04-30 05:23:02,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 34a96274 >2014-04-30 05:23:02,704 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:23:02,725 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-94) [1092d42b] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:23:02,726 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [1092d42b] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3cc3324a >2014-04-30 05:23:02,729 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-94) [1092d42b] FINISH, SetVdsStatusVDSCommand, log id: 3cc3324a >2014-04-30 05:23:02,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [1092d42b] Correlation ID: 1092d42b, Job ID: 60a66e79-84f4-4e55-a498-e0d8b32c50de, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:23:02,751 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [1092d42b] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:23:02,752 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-94) [1092d42b] Correlation ID: 7fb77095, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:23:02,756 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-94) [285748ca] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:23:02,758 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-94) [30e0562] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:23:02,758 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-94) [30e0562] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-17) [2080472c] Autorecovering 1 hosts >2014-04-30 05:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-17) [2080472c] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:25:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: b1d17ef >2014-04-30 05:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] FINISH, SetVdsStatusVDSCommand, log id: b1d17ef >2014-04-30 05:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 449f38c9 >2014-04-30 05:25:00,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3cd82041 >2014-04-30 05:25:00,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] Failed in GetHardwareInfoVDS method >2014-04-30 05:25:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:25:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@112daa9c >2014-04-30 05:25:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] HostName = 10.34.27.209 >2014-04-30 05:25:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:25:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-17) [71506b6d] FINISH, GetHardwareInfoVDSCommand, log id: 3cd82041 >2014-04-30 05:25:00,742 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-17) [71506b6d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:25:00,755 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-17) [57b24684] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:25:00,756 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-17) [2a1a9409] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:25:00,757 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-17) [2a1a9409] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 449f38c9 >2014-04-30 05:25:00,772 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [2a1a9409] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:25:00,773 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-17) [2a1a9409] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:25:03,109 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6fb1b0f >2014-04-30 05:25:03,330 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] Failed in GetHardwareInfoVDS method >2014-04-30 05:25:03,331 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:25:03,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4fad795d >2014-04-30 05:25:03,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] HostName = 10.34.27.209 >2014-04-30 05:25:03,331 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:25:03,331 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-63) [f57c41e] FINISH, GetHardwareInfoVDSCommand, log id: 6fb1b0f >2014-04-30 05:25:03,335 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [f57c41e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:25:03,346 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [f57c41e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:25:03,568 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-63) [3407f5f5] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:25:03,569 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-63) [5f610e9f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:25:03,575 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-63) [5f610e9f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 546e54fa >2014-04-30 05:28:03,577 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-63) [5f610e9f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:28:03,577 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-63) [5f610e9f] FINISH, ConnectStorageServerVDSCommand, log id: 546e54fa >2014-04-30 05:28:03,577 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-63) [5f610e9f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:28:03,582 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 19de4ce6 >2014-04-30 05:28:04,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 19de4ce6 >2014-04-30 05:28:04,036 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:28:04,072 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-63) [308eb762] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:28:04,073 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [308eb762] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3d0b144a >2014-04-30 05:28:04,082 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-63) [308eb762] FINISH, SetVdsStatusVDSCommand, log id: 3d0b144a >2014-04-30 05:28:04,092 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [308eb762] Correlation ID: 308eb762, Job ID: 7077b739-0ae6-401d-a5fd-99122755b959, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:28:04,099 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [308eb762] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:28:04,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-63) [308eb762] Correlation ID: 3407f5f5, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:28:04,112 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-63) [3d7ce6fa] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:28:04,113 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-63) [472e24] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:28:04,114 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-63) [472e24] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-32) [1edcaf5e] Autorecovering 1 hosts >2014-04-30 05:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-32) [1edcaf5e] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7ca82ac9 >2014-04-30 05:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] FINISH, SetVdsStatusVDSCommand, log id: 7ca82ac9 >2014-04-30 05:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6b8815e3 >2014-04-30 05:30:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1babfc04 >2014-04-30 05:30:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] Failed in GetHardwareInfoVDS method >2014-04-30 05:30:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:30:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@73dfd116 >2014-04-30 05:30:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] HostName = 10.34.27.209 >2014-04-30 05:30:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:30:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [36a7434c] FINISH, GetHardwareInfoVDSCommand, log id: 1babfc04 >2014-04-30 05:30:00,703 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [36a7434c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:30:00,713 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-32) [9c653a2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:30:00,714 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:30:00,715 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6b8815e3 >2014-04-30 05:30:00,728 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:30:00,729 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:30:01,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 759e04c5 >2014-04-30 05:30:01,701 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] Failed in GetHardwareInfoVDS method >2014-04-30 05:30:01,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:30:01,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6b60fb1b >2014-04-30 05:30:01,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] HostName = 10.34.27.209 >2014-04-30 05:30:01,702 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:30:01,702 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [538e01bb] FINISH, GetHardwareInfoVDSCommand, log id: 759e04c5 >2014-04-30 05:30:01,707 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [538e01bb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:30:01,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [538e01bb] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:30:01,937 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-64) [1133c551] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:30:01,939 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [7c8596cf] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:30:01,947 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [7c8596cf] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6f2bdd13 >2014-04-30 05:33:01,949 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [7c8596cf] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:33:01,949 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [7c8596cf] FINISH, ConnectStorageServerVDSCommand, log id: 6f2bdd13 >2014-04-30 05:33:01,949 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [7c8596cf] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:33:01,953 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 31fa7078 >2014-04-30 05:33:02,395 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 31fa7078 >2014-04-30 05:33:02,395 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:33:02,425 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-64) [75ac9335] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:33:02,426 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [75ac9335] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1e8c9733 >2014-04-30 05:33:02,437 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [75ac9335] FINISH, SetVdsStatusVDSCommand, log id: 1e8c9733 >2014-04-30 05:33:02,450 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [75ac9335] Correlation ID: 75ac9335, Job ID: 72efac47-3a05-498e-b77c-c52fe452fb79, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:33:02,459 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [75ac9335] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:33:02,465 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [75ac9335] Correlation ID: 1133c551, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:33:02,476 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [51f5482c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:33:02,477 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [1652f88e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:33:02,478 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [1652f88e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-68) [3e1ca74a] Autorecovering 1 hosts >2014-04-30 05:35:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-68) [3e1ca74a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:35:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:35:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 35dc8cd9 >2014-04-30 05:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] FINISH, SetVdsStatusVDSCommand, log id: 35dc8cd9 >2014-04-30 05:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 71e5d09a >2014-04-30 05:35:00,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 27396093 >2014-04-30 05:35:00,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] Failed in GetHardwareInfoVDS method >2014-04-30 05:35:00,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:35:00,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@27f36703 >2014-04-30 05:35:00,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] HostName = 10.34.27.209 >2014-04-30 05:35:00,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:35:00,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [5664e52c] FINISH, GetHardwareInfoVDSCommand, log id: 27396093 >2014-04-30 05:35:00,728 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [5664e52c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:35:00,737 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [232396db] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:35:00,738 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:35:00,739 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 71e5d09a >2014-04-30 05:35:00,752 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:35:00,753 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:35:02,844 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3d2f978e >2014-04-30 05:35:03,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] Failed in GetHardwareInfoVDS method >2014-04-30 05:35:03,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:35:03,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7f870b03 >2014-04-30 05:35:03,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] HostName = 10.34.27.209 >2014-04-30 05:35:03,067 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:35:03,067 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-28) [6611f266] FINISH, GetHardwareInfoVDSCommand, log id: 3d2f978e >2014-04-30 05:35:03,071 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [6611f266] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:35:03,080 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [6611f266] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:35:03,301 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-28) [55e09645] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:35:03,303 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-28) [1a3c51df] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:35:03,309 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [1a3c51df] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 9cdce5b >2014-04-30 05:36:05,023 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-85) [2ad53db9] Start refreshing all users data >2014-04-30 05:36:07,308 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Attempting to update VMs/Templates Ovf. >2014-04-30 05:36:07,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Attempting to update VM OVFs in Data Center NFS >2014-04-30 05:36:07,309 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Successfully updated VM OVFs in Data Center NFS >2014-04-30 05:36:07,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Attempting to update template OVFs in Data Center NFS >2014-04-30 05:36:07,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Successfully updated templates OVFs in Data Center NFS >2014-04-30 05:36:07,310 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-30 05:36:07,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-30 05:38:03,311 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [1a3c51df] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:38:03,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-28) [1a3c51df] FINISH, ConnectStorageServerVDSCommand, log id: 9cdce5b >2014-04-30 05:38:03,312 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-28) [1a3c51df] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:38:03,321 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 71b8379f >2014-04-30 05:38:03,763 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 71b8379f >2014-04-30 05:38:03,764 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:38:03,774 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-28) [48ae06ee] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:38:03,775 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [48ae06ee] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 2daa966f >2014-04-30 05:38:03,777 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-28) [48ae06ee] FINISH, SetVdsStatusVDSCommand, log id: 2daa966f >2014-04-30 05:38:03,783 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [48ae06ee] Correlation ID: 48ae06ee, Job ID: a08558a9-dca0-4bc5-b56a-1c6baec1ebcc, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:38:03,790 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [48ae06ee] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:38:03,792 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-28) [48ae06ee] Correlation ID: 55e09645, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:38:03,797 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-28) [4cff1a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:38:03,798 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-28) [7d7b626d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:38:03,799 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-28) [7d7b626d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [74644392] Autorecovering 1 hosts >2014-04-30 05:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-65) [74644392] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:40:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 39cebb00 >2014-04-30 05:40:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] FINISH, SetVdsStatusVDSCommand, log id: 39cebb00 >2014-04-30 05:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 202283ab >2014-04-30 05:40:01,052 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 78fda95b >2014-04-30 05:40:01,568 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] Failed in GetHardwareInfoVDS method >2014-04-30 05:40:01,568 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:40:01,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5eda659f >2014-04-30 05:40:01,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] HostName = 10.34.27.209 >2014-04-30 05:40:01,569 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:40:01,569 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] FINISH, GetHardwareInfoVDSCommand, log id: 78fda95b >2014-04-30 05:40:01,574 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-65) [1fe3cad2] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:40:01,583 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-65) [379a8b8d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:40:01,585 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-65) [14172e30] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:40:01,585 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-65) [14172e30] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 202283ab >2014-04-30 05:40:01,601 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [14172e30] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:40:01,602 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-65) [14172e30] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:40:04,374 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 698d8036 >2014-04-30 05:40:04,833 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] Failed in GetHardwareInfoVDS method >2014-04-30 05:40:04,833 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:40:04,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2e079264 >2014-04-30 05:40:04,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] HostName = 10.34.27.209 >2014-04-30 05:40:04,833 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:40:04,833 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] FINISH, GetHardwareInfoVDSCommand, log id: 698d8036 >2014-04-30 05:40:04,846 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:40:04,878 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [2d9e55b8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:40:05,354 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-7) [3f29aa5a] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:40:05,356 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [5a605a09] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:40:05,363 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [5a605a09] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 61a09ae6 >2014-04-30 05:43:05,364 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [5a605a09] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:43:05,365 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [5a605a09] FINISH, ConnectStorageServerVDSCommand, log id: 61a09ae6 >2014-04-30 05:43:05,365 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [5a605a09] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:43:05,371 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 243e37b0 >2014-04-30 05:43:05,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 243e37b0 >2014-04-30 05:43:05,814 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:43:05,826 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-7) [2ce0d788] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:43:05,827 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [2ce0d788] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 51935936 >2014-04-30 05:43:05,829 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [2ce0d788] FINISH, SetVdsStatusVDSCommand, log id: 51935936 >2014-04-30 05:43:05,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [2ce0d788] Correlation ID: 2ce0d788, Job ID: 5fca5231-f889-4a48-826a-ce7fea446ff1, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:43:05,841 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [2ce0d788] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:43:05,843 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [2ce0d788] Correlation ID: 3f29aa5a, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:43:05,849 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-7) [71a4fc41] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:43:05,850 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [28f8ef11] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:43:05,850 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [28f8ef11] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:45:00,031 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-43) [20e5daa5] Autorecovering 1 hosts >2014-04-30 05:45:00,031 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-43) [20e5daa5] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:45:00,032 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:45:00,032 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:45:00,032 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:45:00,032 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:45:00,037 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 52887518 >2014-04-30 05:45:00,040 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] FINISH, SetVdsStatusVDSCommand, log id: 52887518 >2014-04-30 05:45:00,042 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6d6436b0 >2014-04-30 05:45:00,510 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 10ca512 >2014-04-30 05:45:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] Failed in GetHardwareInfoVDS method >2014-04-30 05:45:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:45:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@65294147 >2014-04-30 05:45:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] HostName = 10.34.27.209 >2014-04-30 05:45:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:45:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [2895ed94] FINISH, GetHardwareInfoVDSCommand, log id: 10ca512 >2014-04-30 05:45:00,738 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [2895ed94] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:45:00,748 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [5e0793ae] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:45:00,749 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:45:00,749 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6d6436b0 >2014-04-30 05:45:00,762 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:45:00,763 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:45:03,228 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7ffad043] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: b13dbc9 >2014-04-30 05:45:03,451 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7ffad043] Failed in GetHardwareInfoVDS method >2014-04-30 05:45:03,451 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7ffad043] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:45:03,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7ffad043] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@44c26ecd >2014-04-30 05:45:03,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7ffad043] HostName = 10.34.27.209 >2014-04-30 05:45:03,451 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7ffad043] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:45:03,451 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-19) [7ffad043] FINISH, GetHardwareInfoVDSCommand, log id: b13dbc9 >2014-04-30 05:45:03,456 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [7ffad043] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:45:03,465 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [7ffad043] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:45:03,686 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-19) [3f0fc52] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:45:03,687 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [5913d8d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:45:03,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [5913d8d3] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 69ca1b77 >2014-04-30 05:48:03,699 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [5913d8d3] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:48:03,699 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-19) [5913d8d3] FINISH, ConnectStorageServerVDSCommand, log id: 69ca1b77 >2014-04-30 05:48:03,699 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-19) [5913d8d3] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:48:03,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 6fa2b5a1 >2014-04-30 05:48:04,151 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 6fa2b5a1 >2014-04-30 05:48:04,151 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:48:04,162 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-19) [93df968] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:48:04,163 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [93df968] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5d866cb9 >2014-04-30 05:48:04,166 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-19) [93df968] FINISH, SetVdsStatusVDSCommand, log id: 5d866cb9 >2014-04-30 05:48:04,173 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [93df968] Correlation ID: 93df968, Job ID: a6956900-90a6-492f-8297-c4e092bb8a11, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:48:04,180 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [93df968] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:48:04,182 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-19) [93df968] Correlation ID: 3f0fc52, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:48:04,186 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-19) [4c1a501f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:48:04,187 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-19) [6e529d5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:48:04,188 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-19) [6e529d5] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:50:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) [2d164364] Autorecovering 1 hosts >2014-04-30 05:50:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) [2d164364] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:50:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:50:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:50:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:50:00,009 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:50:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5678830d >2014-04-30 05:50:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] FINISH, SetVdsStatusVDSCommand, log id: 5678830d >2014-04-30 05:50:00,019 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3472760f >2014-04-30 05:50:00,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3276144c >2014-04-30 05:50:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] Failed in GetHardwareInfoVDS method >2014-04-30 05:50:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:50:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@662ea375 >2014-04-30 05:50:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] HostName = 10.34.27.209 >2014-04-30 05:50:00,705 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:50:00,705 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [2abba7c0] FINISH, GetHardwareInfoVDSCommand, log id: 3276144c >2014-04-30 05:50:00,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [2abba7c0] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:50:00,727 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-51) [66b4abf9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:50:00,728 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-51) [1dc0605] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:50:00,729 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) [1dc0605] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3472760f >2014-04-30 05:50:00,744 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [1dc0605] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:50:00,745 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [1dc0605] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:50:01,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [409cceea] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: f8b5b7 >2014-04-30 05:50:01,758 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [409cceea] Failed in GetHardwareInfoVDS method >2014-04-30 05:50:01,758 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [409cceea] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:50:01,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [409cceea] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@42277625 >2014-04-30 05:50:01,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [409cceea] HostName = 10.34.27.209 >2014-04-30 05:50:01,758 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [409cceea] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:50:01,758 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [409cceea] FINISH, GetHardwareInfoVDSCommand, log id: f8b5b7 >2014-04-30 05:50:01,764 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [409cceea] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:50:01,775 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [409cceea] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:50:01,996 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-45) [2b492a97] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:50:01,998 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [7c580090] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:50:02,007 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [7c580090] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6f390c5 >2014-04-30 05:53:02,009 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [7c580090] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:53:02,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [7c580090] FINISH, ConnectStorageServerVDSCommand, log id: 6f390c5 >2014-04-30 05:53:02,009 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [7c580090] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:53:02,020 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7477f685 >2014-04-30 05:53:02,477 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 7477f685 >2014-04-30 05:53:02,477 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:53:02,487 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-45) [429f36a4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:53:02,488 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [429f36a4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 61d6fb53 >2014-04-30 05:53:02,490 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [429f36a4] FINISH, SetVdsStatusVDSCommand, log id: 61d6fb53 >2014-04-30 05:53:02,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [429f36a4] Correlation ID: 429f36a4, Job ID: 238730c6-e864-450d-b207-6da9ef03328c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:53:02,506 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [429f36a4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:53:02,508 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [429f36a4] Correlation ID: 2b492a97, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:53:02,511 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [3adbd1b9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:53:02,512 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [42b0efd0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:53:02,513 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) [42b0efd0] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 05:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-92) [6b8f6d0c] Autorecovering 1 hosts >2014-04-30 05:55:00,002 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-92) [6b8f6d0c] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 05:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [69620189] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [69620189] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [69620189] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 05:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [69620189] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 05:55:00,006 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [69620189] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 37ef0e83 >2014-04-30 05:55:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [69620189] FINISH, SetVdsStatusVDSCommand, log id: 37ef0e83 >2014-04-30 05:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) [69620189] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 74555510 >2014-04-30 05:55:00,476 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [69620189] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4929356a >2014-04-30 05:55:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [69620189] Failed in GetHardwareInfoVDS method >2014-04-30 05:55:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [69620189] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:55:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [69620189] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3071e7ac >2014-04-30 05:55:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [69620189] HostName = 10.34.27.209 >2014-04-30 05:55:00,698 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [69620189] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:55:00,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [69620189] FINISH, GetHardwareInfoVDSCommand, log id: 4929356a >2014-04-30 05:55:00,703 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [69620189] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:55:00,713 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-92) [243719b7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:55:00,714 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-92) [3b3dfd08] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:55:00,715 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) [3b3dfd08] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 74555510 >2014-04-30 05:55:00,728 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [3b3dfd08] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 05:55:00,729 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [3b3dfd08] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 05:55:02,868 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [24676a68] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 699989c0 >2014-04-30 05:55:03,092 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [24676a68] Failed in GetHardwareInfoVDS method >2014-04-30 05:55:03,092 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [24676a68] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:55:03,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [24676a68] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5f22d269 >2014-04-30 05:55:03,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [24676a68] HostName = 10.34.27.209 >2014-04-30 05:55:03,092 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [24676a68] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 05:55:03,092 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [24676a68] FINISH, GetHardwareInfoVDSCommand, log id: 699989c0 >2014-04-30 05:55:03,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [24676a68] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 05:55:03,108 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [24676a68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 05:55:03,330 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-55) [5c6f537c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:55:03,331 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [4c02740a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 05:55:03,339 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [4c02740a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2a00bc96 >2014-04-30 05:58:03,341 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [4c02740a] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 05:58:03,341 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [4c02740a] FINISH, ConnectStorageServerVDSCommand, log id: 2a00bc96 >2014-04-30 05:58:03,341 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [4c02740a] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 05:58:03,350 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 6654ce2f >2014-04-30 05:58:03,801 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 6654ce2f >2014-04-30 05:58:03,801 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 05:58:03,811 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-55) [78acc2e4] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:58:03,812 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [78acc2e4] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 54461b5e >2014-04-30 05:58:03,814 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [78acc2e4] FINISH, SetVdsStatusVDSCommand, log id: 54461b5e >2014-04-30 05:58:03,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [78acc2e4] Correlation ID: 78acc2e4, Job ID: b2e8c597-b50e-4e37-a187-ac72dde0ed32, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 05:58:03,826 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [78acc2e4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 05:58:03,827 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [78acc2e4] Correlation ID: 5c6f537c, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 05:58:03,831 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [31eea2b0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:58:03,832 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 05:58:03,832 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-89) [7b33c831] Autorecovering 1 hosts >2014-04-30 06:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-89) [7b33c831] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3e5a8a0e >2014-04-30 06:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] FINISH, SetVdsStatusVDSCommand, log id: 3e5a8a0e >2014-04-30 06:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 58a3d0a8 >2014-04-30 06:00:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1d95ff74 >2014-04-30 06:00:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] Failed in GetHardwareInfoVDS method >2014-04-30 06:00:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:00:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@62b627d8 >2014-04-30 06:00:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] HostName = 10.34.27.209 >2014-04-30 06:00:00,697 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:00:00,697 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-89) [42ab3472] FINISH, GetHardwareInfoVDSCommand, log id: 1d95ff74 >2014-04-30 06:00:00,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-89) [42ab3472] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:00:00,733 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-89) [554955a0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:00:00,734 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-89) [5f4968da] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:00:00,735 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-89) [5f4968da] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 58a3d0a8 >2014-04-30 06:00:00,750 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [5f4968da] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:00:00,751 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-89) [5f4968da] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:00:01,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 20a6c454 >2014-04-30 06:00:01,409 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] Failed in GetHardwareInfoVDS method >2014-04-30 06:00:01,410 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:00:01,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@24a3778b >2014-04-30 06:00:01,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] HostName = 10.34.27.209 >2014-04-30 06:00:01,410 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:00:01,410 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [3f366169] FINISH, GetHardwareInfoVDSCommand, log id: 20a6c454 >2014-04-30 06:00:01,414 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [3f366169] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:00:01,423 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [3f366169] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:00:01,648 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-73) [51206f67] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:00:01,650 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [bfd2bee] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:00:01,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [bfd2bee] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6182aa34 >2014-04-30 06:03:01,659 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [bfd2bee] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:03:01,659 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [bfd2bee] FINISH, ConnectStorageServerVDSCommand, log id: 6182aa34 >2014-04-30 06:03:01,659 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [bfd2bee] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:03:01,663 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 3fa5bbe5 >2014-04-30 06:03:02,102 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 3fa5bbe5 >2014-04-30 06:03:02,102 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:03:02,113 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-73) [2e871787] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:03:02,114 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [2e871787] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 152d5966 >2014-04-30 06:03:02,116 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [2e871787] FINISH, SetVdsStatusVDSCommand, log id: 152d5966 >2014-04-30 06:03:02,123 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [2e871787] Correlation ID: 2e871787, Job ID: 4e1a733e-0ab1-4da1-a617-0ab39bd3cee8, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:03:02,143 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [2e871787] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:03:02,145 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [2e871787] Correlation ID: 51206f67, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:03:02,149 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [55256248] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:03:02,150 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:03:02,151 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-82) [24d883ab] Autorecovering 1 hosts >2014-04-30 06:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-82) [24d883ab] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 52d3d73c >2014-04-30 06:05:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] FINISH, SetVdsStatusVDSCommand, log id: 52d3d73c >2014-04-30 06:05:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4cefc15a >2014-04-30 06:05:00,491 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7e659a61 >2014-04-30 06:05:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] Failed in GetHardwareInfoVDS method >2014-04-30 06:05:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:05:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@407a971b >2014-04-30 06:05:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] HostName = 10.34.27.209 >2014-04-30 06:05:00,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:05:00,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [c4fffe1] FINISH, GetHardwareInfoVDSCommand, log id: 7e659a61 >2014-04-30 06:05:00,720 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [c4fffe1] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:05:00,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-82) [45f6f206] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:05:00,730 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-82) [2830d87f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:05:00,734 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) [2830d87f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4cefc15a >2014-04-30 06:05:00,757 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [2830d87f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:05:00,758 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [2830d87f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:05:02,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [33f00ca3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6e410891 >2014-04-30 06:05:02,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Failed in GetHardwareInfoVDS method >2014-04-30 06:05:02,723 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:05:02,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6f99a023 >2014-04-30 06:05:02,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [33f00ca3] HostName = 10.34.27.209 >2014-04-30 06:05:02,724 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:05:02,724 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-12) [33f00ca3] FINISH, GetHardwareInfoVDSCommand, log id: 6e410891 >2014-04-30 06:05:02,729 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:05:02,737 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [33f00ca3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:05:02,961 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-12) [4a7365ef] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:05:02,963 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [6f143806] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:05:02,970 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [6f143806] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4fa01387 >2014-04-30 06:08:02,971 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [6f143806] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:08:02,971 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-12) [6f143806] FINISH, ConnectStorageServerVDSCommand, log id: 4fa01387 >2014-04-30 06:08:02,972 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-12) [6f143806] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:08:02,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7f2351bf >2014-04-30 06:08:03,422 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 7f2351bf >2014-04-30 06:08:03,422 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:08:03,433 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-12) [4ffb3267] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:08:03,434 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [4ffb3267] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 41376bb1 >2014-04-30 06:08:03,436 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-12) [4ffb3267] FINISH, SetVdsStatusVDSCommand, log id: 41376bb1 >2014-04-30 06:08:03,442 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [4ffb3267] Correlation ID: 4ffb3267, Job ID: feaff2d6-33c5-4c97-84b6-6afacf53f188, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:08:03,449 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [4ffb3267] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:08:03,450 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-12) [4ffb3267] Correlation ID: 4a7365ef, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:08:03,455 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-12) [28a8e4ce] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:08:03,456 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-12) [1abb3c2a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:08:03,457 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-12) [1abb3c2a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) [43b2e52a] Autorecovering 1 hosts >2014-04-30 06:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) [43b2e52a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:10:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:10:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5433efa2 >2014-04-30 06:10:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] FINISH, SetVdsStatusVDSCommand, log id: 5433efa2 >2014-04-30 06:10:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 43d9c9e2 >2014-04-30 06:10:00,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 8b68145 >2014-04-30 06:10:00,731 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] Failed in GetHardwareInfoVDS method >2014-04-30 06:10:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:10:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6f1d42e8 >2014-04-30 06:10:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] HostName = 10.34.27.209 >2014-04-30 06:10:00,732 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:10:00,732 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [56377e2d] FINISH, GetHardwareInfoVDSCommand, log id: 8b68145 >2014-04-30 06:10:00,737 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [56377e2d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:10:00,745 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-54) [1408c8e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:10:00,746 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-54) [1d5f1b42] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:10:00,747 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) [1d5f1b42] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 43d9c9e2 >2014-04-30 06:10:00,772 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [1d5f1b42] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:10:00,773 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [1d5f1b42] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:10:03,823 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [64b2333f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6eb2508a >2014-04-30 06:10:04,045 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [64b2333f] Failed in GetHardwareInfoVDS method >2014-04-30 06:10:04,046 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [64b2333f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:10:04,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [64b2333f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1e08a513 >2014-04-30 06:10:04,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [64b2333f] HostName = 10.34.27.209 >2014-04-30 06:10:04,046 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [64b2333f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:10:04,046 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [64b2333f] FINISH, GetHardwareInfoVDSCommand, log id: 6eb2508a >2014-04-30 06:10:04,051 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [64b2333f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:10:04,063 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [64b2333f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:10:04,287 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-4) [6ae40de2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:10:04,292 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-4) [46906788] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:10:04,302 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [46906788] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4821ef8c >2014-04-30 06:13:04,304 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [46906788] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:13:04,304 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [46906788] FINISH, ConnectStorageServerVDSCommand, log id: 4821ef8c >2014-04-30 06:13:04,304 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-4) [46906788] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:13:04,311 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 46920824 >2014-04-30 06:13:04,753 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 46920824 >2014-04-30 06:13:04,753 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:13:04,764 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-4) [762902cc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:13:04,765 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [762902cc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 52e7d966 >2014-04-30 06:13:04,768 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [762902cc] FINISH, SetVdsStatusVDSCommand, log id: 52e7d966 >2014-04-30 06:13:04,777 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [762902cc] Correlation ID: 762902cc, Job ID: b07efafa-9fcb-4468-bc0d-c633578a92e3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:13:04,784 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [762902cc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:13:04,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [762902cc] Correlation ID: 6ae40de2, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:13:04,791 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-4) [6f4b4bbf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:13:04,792 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-4) [50fdeab5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:13:04,793 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) [50fdeab5] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) [1dc0605] Autorecovering 1 hosts >2014-04-30 06:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-51) [1dc0605] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 18a924 >2014-04-30 06:15:00,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] FINISH, SetVdsStatusVDSCommand, log id: 18a924 >2014-04-30 06:15:00,020 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1367cad >2014-04-30 06:15:00,480 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2d3fbbf3 >2014-04-30 06:15:00,703 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] Failed in GetHardwareInfoVDS method >2014-04-30 06:15:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:15:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@334c65ba >2014-04-30 06:15:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] HostName = 10.34.27.209 >2014-04-30 06:15:00,704 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:15:00,704 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [60831e12] FINISH, GetHardwareInfoVDSCommand, log id: 2d3fbbf3 >2014-04-30 06:15:00,709 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [60831e12] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:15:00,719 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-51) [2ae4d73f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:15:00,720 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:15:00,721 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1367cad >2014-04-30 06:15:00,752 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:15:00,753 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:15:02,183 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [51c4293] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6dc5779d >2014-04-30 06:15:02,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [51c4293] Failed in GetHardwareInfoVDS method >2014-04-30 06:15:02,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [51c4293] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:15:02,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [51c4293] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3f301cb8 >2014-04-30 06:15:02,407 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [51c4293] HostName = 10.34.27.209 >2014-04-30 06:15:02,407 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [51c4293] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:15:02,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [51c4293] FINISH, GetHardwareInfoVDSCommand, log id: 6dc5779d >2014-04-30 06:15:02,413 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [51c4293] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:15:02,425 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [51c4293] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:15:02,650 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-81) [2efbd9b1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:15:02,651 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [3feab3d1] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:15:02,658 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [3feab3d1] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 790973e5 >2014-04-30 06:18:02,660 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [3feab3d1] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:18:02,660 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-81) [3feab3d1] FINISH, ConnectStorageServerVDSCommand, log id: 790973e5 >2014-04-30 06:18:02,660 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-81) [3feab3d1] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:18:02,666 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 581d8430 >2014-04-30 06:18:03,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 581d8430 >2014-04-30 06:18:03,117 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:18:03,130 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-81) [5563f1e2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:18:03,131 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [5563f1e2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4bc06ed4 >2014-04-30 06:18:03,134 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [5563f1e2] FINISH, SetVdsStatusVDSCommand, log id: 4bc06ed4 >2014-04-30 06:18:03,139 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [5563f1e2] Correlation ID: 5563f1e2, Job ID: c02ace18-8352-4b7e-9358-a40295c0ebf2, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:18:03,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [5563f1e2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:18:03,148 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [5563f1e2] Correlation ID: 2efbd9b1, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:18:03,152 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [7dc4b70a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:18:03,153 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [429b86b3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:18:03,154 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-81) [429b86b3] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-92) [3b3dfd08] Autorecovering 1 hosts >2014-04-30 06:20:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-92) [3b3dfd08] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:20:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:20:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:20:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:20:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3988be9e >2014-04-30 06:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] FINISH, SetVdsStatusVDSCommand, log id: 3988be9e >2014-04-30 06:20:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 361aa7e0 >2014-04-30 06:20:00,478 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 55806e58 >2014-04-30 06:20:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] Failed in GetHardwareInfoVDS method >2014-04-30 06:20:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:20:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@62cc7f42 >2014-04-30 06:20:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] HostName = 10.34.27.209 >2014-04-30 06:20:00,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:20:00,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [16adac7c] FINISH, GetHardwareInfoVDSCommand, log id: 55806e58 >2014-04-30 06:20:00,706 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [16adac7c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:20:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-92) [6f34623f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:20:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-92) [22bf197f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:20:00,717 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) [22bf197f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 361aa7e0 >2014-04-30 06:20:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [22bf197f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:20:00,734 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [22bf197f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:20:03,501 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7928dd51 >2014-04-30 06:20:03,726 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] Failed in GetHardwareInfoVDS method >2014-04-30 06:20:03,726 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:20:03,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5480d6a3 >2014-04-30 06:20:03,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] HostName = 10.34.27.209 >2014-04-30 06:20:03,726 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:20:03,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-52) [30c72d1f] FINISH, GetHardwareInfoVDSCommand, log id: 7928dd51 >2014-04-30 06:20:03,731 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [30c72d1f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:20:03,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [30c72d1f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:20:03,963 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-52) [5ac58543] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:20:03,965 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-52) [5727cdca] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:20:03,974 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [5727cdca] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 630ebd01 >2014-04-30 06:23:03,976 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [5727cdca] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:23:03,976 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-52) [5727cdca] FINISH, ConnectStorageServerVDSCommand, log id: 630ebd01 >2014-04-30 06:23:03,976 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-52) [5727cdca] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:23:03,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 362f6027 >2014-04-30 06:23:04,453 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 362f6027 >2014-04-30 06:23:04,453 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:23:04,468 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-52) [25601664] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:23:04,470 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [25601664] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 586acc92 >2014-04-30 06:23:04,476 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-52) [25601664] FINISH, SetVdsStatusVDSCommand, log id: 586acc92 >2014-04-30 06:23:04,483 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [25601664] Correlation ID: 25601664, Job ID: 9fca3657-bf57-4e76-badc-ccb6c91d97fa, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:23:04,493 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [25601664] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:23:04,495 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-52) [25601664] Correlation ID: 5ac58543, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:23:04,503 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-52) [42fa450e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:23:04,504 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-52) [55ea1fd9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:23:04,505 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-52) [55ea1fd9] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) [1718c2d] Autorecovering 1 hosts >2014-04-30 06:25:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) [1718c2d] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:25:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3db3024a >2014-04-30 06:25:00,016 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] FINISH, SetVdsStatusVDSCommand, log id: 3db3024a >2014-04-30 06:25:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 71f453d0 >2014-04-30 06:25:00,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6a68e96d >2014-04-30 06:25:00,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] Failed in GetHardwareInfoVDS method >2014-04-30 06:25:00,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:25:00,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@db529af >2014-04-30 06:25:00,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] HostName = 10.34.27.209 >2014-04-30 06:25:00,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:25:00,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [5c745161] FINISH, GetHardwareInfoVDSCommand, log id: 6a68e96d >2014-04-30 06:25:00,743 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [5c745161] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:25:00,752 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [7054e306] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:25:00,753 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [7b54e090] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:25:00,754 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [7b54e090] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 71f453d0 >2014-04-30 06:25:00,767 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [7b54e090] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:25:00,768 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [7b54e090] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:25:01,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [7337d50d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7476721b >2014-04-30 06:25:02,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [7337d50d] Failed in GetHardwareInfoVDS method >2014-04-30 06:25:02,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [7337d50d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:25:02,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [7337d50d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@294962b1 >2014-04-30 06:25:02,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [7337d50d] HostName = 10.34.27.209 >2014-04-30 06:25:02,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [7337d50d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:25:02,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-22) [7337d50d] FINISH, GetHardwareInfoVDSCommand, log id: 7476721b >2014-04-30 06:25:02,104 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [7337d50d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:25:02,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [7337d50d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:25:02,348 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-22) [5e5a184c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:25:02,350 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [5b6afa13] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:25:02,356 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [5b6afa13] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 9b41c58 >2014-04-30 06:28:02,357 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [5b6afa13] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:28:02,357 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-22) [5b6afa13] FINISH, ConnectStorageServerVDSCommand, log id: 9b41c58 >2014-04-30 06:28:02,357 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-22) [5b6afa13] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:28:02,362 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 48f202e2 >2014-04-30 06:28:02,824 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 48f202e2 >2014-04-30 06:28:02,824 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:28:02,836 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-22) [412f599a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:28:02,837 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [412f599a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 46d6856e >2014-04-30 06:28:02,840 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-22) [412f599a] FINISH, SetVdsStatusVDSCommand, log id: 46d6856e >2014-04-30 06:28:02,847 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [412f599a] Correlation ID: 412f599a, Job ID: c093d8b8-f43c-4638-99fd-c943a26869e6, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:28:02,855 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [412f599a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:28:02,857 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-22) [412f599a] Correlation ID: 5e5a184c, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:28:02,861 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-22) [20fc1763] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:28:02,862 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-22) [40945c72] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:28:02,863 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-22) [40945c72] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-82) [2830d87f] Autorecovering 1 hosts >2014-04-30 06:30:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-82) [2830d87f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:30:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 573211d5 >2014-04-30 06:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] FINISH, SetVdsStatusVDSCommand, log id: 573211d5 >2014-04-30 06:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 434af8ba >2014-04-30 06:30:00,506 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: b907404 >2014-04-30 06:30:00,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] Failed in GetHardwareInfoVDS method >2014-04-30 06:30:00,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:30:00,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5a238919 >2014-04-30 06:30:00,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] HostName = 10.34.27.209 >2014-04-30 06:30:00,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:30:00,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-82) [84e6f78] FINISH, GetHardwareInfoVDSCommand, log id: b907404 >2014-04-30 06:30:00,744 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [84e6f78] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:30:00,756 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-82) [5b589282] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:30:00,757 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-82) [573577a4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:30:00,758 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-82) [573577a4] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 434af8ba >2014-04-30 06:30:00,772 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [573577a4] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:30:00,775 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-82) [573577a4] Correlation ID: 84e6f78, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 was autorecovered. >2014-04-30 06:30:00,775 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-82) [573577a4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:30:03,236 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [48585b8e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 11e845d3 >2014-04-30 06:30:03,469 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [48585b8e] Failed in GetHardwareInfoVDS method >2014-04-30 06:30:03,469 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [48585b8e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:30:03,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [48585b8e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@23fdc884 >2014-04-30 06:30:03,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [48585b8e] HostName = 10.34.27.209 >2014-04-30 06:30:03,469 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [48585b8e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:30:03,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-84) [48585b8e] FINISH, GetHardwareInfoVDSCommand, log id: 11e845d3 >2014-04-30 06:30:03,476 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [48585b8e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:30:03,485 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [48585b8e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:30:03,717 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-84) [7096891d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:30:03,719 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [5feeb86b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:30:03,726 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [5feeb86b] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 50e600b8 >2014-04-30 06:33:03,727 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [5feeb86b] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:33:03,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-84) [5feeb86b] FINISH, ConnectStorageServerVDSCommand, log id: 50e600b8 >2014-04-30 06:33:03,727 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-84) [5feeb86b] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:33:03,733 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 13976777 >2014-04-30 06:33:04,202 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 13976777 >2014-04-30 06:33:04,202 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:33:04,214 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-84) [75cfde5c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:33:04,216 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [75cfde5c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1280690f >2014-04-30 06:33:04,219 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-84) [75cfde5c] FINISH, SetVdsStatusVDSCommand, log id: 1280690f >2014-04-30 06:33:04,225 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [75cfde5c] Correlation ID: 75cfde5c, Job ID: c9b68428-6f5b-42a2-8030-b7d098a4b1e1, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:33:04,232 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [75cfde5c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:33:04,235 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-84) [75cfde5c] Correlation ID: 7096891d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:33:04,240 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-84) [476c09fd] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:33:04,241 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-84) [4ad47998] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:33:04,242 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-84) [4ad47998] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [794dcbc1] Autorecovering 1 hosts >2014-04-30 06:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [794dcbc1] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 58608525 >2014-04-30 06:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] FINISH, SetVdsStatusVDSCommand, log id: 58608525 >2014-04-30 06:35:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 686b9a8c >2014-04-30 06:35:00,497 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6cb15081 >2014-04-30 06:35:00,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] Failed in GetHardwareInfoVDS method >2014-04-30 06:35:00,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:35:00,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@468791be >2014-04-30 06:35:00,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] HostName = 10.34.27.209 >2014-04-30 06:35:00,729 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:35:00,729 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [736e1e8d] FINISH, GetHardwareInfoVDSCommand, log id: 6cb15081 >2014-04-30 06:35:00,734 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [736e1e8d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:35:00,743 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [3a37923a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:35:00,745 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [49ae3be3] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:35:00,746 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [49ae3be3] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 686b9a8c >2014-04-30 06:35:00,765 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [49ae3be3] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:35:00,766 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [49ae3be3] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:35:01,613 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 62c78946 >2014-04-30 06:35:01,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] Failed in GetHardwareInfoVDS method >2014-04-30 06:35:01,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:35:01,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@895a9d9 >2014-04-30 06:35:01,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] HostName = 10.34.27.209 >2014-04-30 06:35:01,845 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:35:01,845 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] FINISH, GetHardwareInfoVDSCommand, log id: 62c78946 >2014-04-30 06:35:01,850 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:35:01,860 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [5c94cdaa] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:35:02,098 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-96) [4e8ded47] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:35:02,100 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-96) [5172a63c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:35:02,106 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [5172a63c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 78a5e439 >2014-04-30 06:36:05,024 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-33) [2c238432] Start refreshing all users data >2014-04-30 06:36:07,311 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [49239c0f] Attempting to update VMs/Templates Ovf. >2014-04-30 06:36:07,312 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [49239c0f] Attempting to update VM OVFs in Data Center NFS >2014-04-30 06:36:07,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [49239c0f] Successfully updated VM OVFs in Data Center NFS >2014-04-30 06:36:07,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [49239c0f] Attempting to update template OVFs in Data Center NFS >2014-04-30 06:36:07,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [49239c0f] Successfully updated templates OVFs in Data Center NFS >2014-04-30 06:36:07,313 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [49239c0f] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-30 06:36:07,314 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-90) [49239c0f] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-30 06:38:02,107 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [5172a63c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:38:02,107 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-96) [5172a63c] FINISH, ConnectStorageServerVDSCommand, log id: 78a5e439 >2014-04-30 06:38:02,107 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-96) [5172a63c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:38:02,111 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7f171d39 >2014-04-30 06:38:02,570 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 7f171d39 >2014-04-30 06:38:02,571 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:38:02,583 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-96) [7bcde031] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:38:02,585 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-96) [7bcde031] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 36cfafe7 >2014-04-30 06:38:02,588 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-96) [7bcde031] FINISH, SetVdsStatusVDSCommand, log id: 36cfafe7 >2014-04-30 06:38:02,594 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [7bcde031] Correlation ID: 7bcde031, Job ID: 614318a3-b80d-4149-8d3a-3971478af1da, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:38:02,602 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [7bcde031] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:38:02,604 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [7bcde031] Correlation ID: 4e8ded47, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:38:02,608 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-96) [5d0a391e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:38:02,609 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-96) [173d021a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:38:02,611 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-96) [173d021a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) [399037c0] Autorecovering 1 hosts >2014-04-30 06:40:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-14) [399037c0] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2b858a16 >2014-04-30 06:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] FINISH, SetVdsStatusVDSCommand, log id: 2b858a16 >2014-04-30 06:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 618cb8e >2014-04-30 06:40:00,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 61fbdc4 >2014-04-30 06:40:00,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] Failed in GetHardwareInfoVDS method >2014-04-30 06:40:00,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:40:00,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@de39200 >2014-04-30 06:40:00,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] HostName = 10.34.27.209 >2014-04-30 06:40:00,736 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:40:00,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-14) [6054fb00] FINISH, GetHardwareInfoVDSCommand, log id: 61fbdc4 >2014-04-30 06:40:00,741 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-14) [6054fb00] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:40:00,750 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-14) [12a4157d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:40:00,751 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-14) [4bdb1a0a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:40:00,752 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-14) [4bdb1a0a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 618cb8e >2014-04-30 06:40:00,770 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [4bdb1a0a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:40:00,771 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-14) [4bdb1a0a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:40:02,993 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 21a76ae1 >2014-04-30 06:40:03,226 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] Failed in GetHardwareInfoVDS method >2014-04-30 06:40:03,226 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:40:03,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@54315a3c >2014-04-30 06:40:03,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] HostName = 10.34.27.209 >2014-04-30 06:40:03,226 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:40:03,226 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] FINISH, GetHardwareInfoVDSCommand, log id: 21a76ae1 >2014-04-30 06:40:03,231 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:40:03,240 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [4cd9a18a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:40:03,475 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-51) [1d4faeb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:40:03,477 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [178f2068] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:40:03,483 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [178f2068] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3339f95 >2014-04-30 06:43:03,485 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [178f2068] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:43:03,485 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-51) [178f2068] FINISH, ConnectStorageServerVDSCommand, log id: 3339f95 >2014-04-30 06:43:03,485 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-51) [178f2068] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:43:03,492 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5471a71c >2014-04-30 06:43:04,525 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5471a71c >2014-04-30 06:43:04,525 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:43:04,535 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-51) [da9913d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:43:04,536 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [da9913d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4767f225 >2014-04-30 06:43:04,539 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-51) [da9913d] FINISH, SetVdsStatusVDSCommand, log id: 4767f225 >2014-04-30 06:43:04,544 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [da9913d] Correlation ID: da9913d, Job ID: f0f002d9-83be-4b72-aeda-c9284d36d848, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:43:04,551 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [da9913d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:43:04,552 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-51) [da9913d] Correlation ID: 1d4faeb, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:43:04,556 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-51) [1ccdb45d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:43:04,557 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-51) [32ae69a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:43:04,558 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-51) [32ae69a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) [5904254f] Autorecovering 1 hosts >2014-04-30 06:45:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-24) [5904254f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [22951063] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [22951063] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [22951063] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [22951063] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:45:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [22951063] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4b988b06 >2014-04-30 06:45:00,041 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-24) [22951063] FINISH, SetVdsStatusVDSCommand, log id: 4b988b06 >2014-04-30 06:45:00,043 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-24) [22951063] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 3f8e980a >2014-04-30 06:45:01,152 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [22951063] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1414f362 >2014-04-30 06:45:01,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [22951063] Failed in GetHardwareInfoVDS method >2014-04-30 06:45:01,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [22951063] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:45:01,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [22951063] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6b76b13a >2014-04-30 06:45:01,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [22951063] HostName = 10.34.27.209 >2014-04-30 06:45:01,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [22951063] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:45:01,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-24) [22951063] FINISH, GetHardwareInfoVDSCommand, log id: 1414f362 >2014-04-30 06:45:01,404 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-24) [22951063] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:45:01,425 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-24) [632a4ed0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:45:01,426 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-24) [472e1d16] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:45:01,427 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-24) [472e1d16] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 3f8e980a >2014-04-30 06:45:01,441 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [472e1d16] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:45:01,442 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-24) [472e1d16] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:45:01,928 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [451e95f6] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 38c6a18f >2014-04-30 06:45:02,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [451e95f6] Failed in GetHardwareInfoVDS method >2014-04-30 06:45:02,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [451e95f6] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:45:02,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [451e95f6] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5a4e84f0 >2014-04-30 06:45:02,223 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [451e95f6] HostName = 10.34.27.209 >2014-04-30 06:45:02,223 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [451e95f6] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:45:02,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-13) [451e95f6] FINISH, GetHardwareInfoVDSCommand, log id: 38c6a18f >2014-04-30 06:45:02,228 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [451e95f6] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:45:02,238 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [451e95f6] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:45:02,569 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-13) [7612f9cb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:45:02,570 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-13) [4f16617f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:45:02,578 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [4f16617f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6b89752c >2014-04-30 06:48:02,580 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [4f16617f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:48:02,580 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-13) [4f16617f] FINISH, ConnectStorageServerVDSCommand, log id: 6b89752c >2014-04-30 06:48:02,581 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-13) [4f16617f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:48:02,586 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 78262180 >2014-04-30 06:48:03,056 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 78262180 >2014-04-30 06:48:03,056 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:48:03,067 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-13) [666eb784] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:48:03,068 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [666eb784] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 6760e47b >2014-04-30 06:48:03,070 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-13) [666eb784] FINISH, SetVdsStatusVDSCommand, log id: 6760e47b >2014-04-30 06:48:03,077 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [666eb784] Correlation ID: 666eb784, Job ID: adce4e28-2a5e-4671-9ac7-b7d619154684, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:48:03,083 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [666eb784] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:48:03,085 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-13) [666eb784] Correlation ID: 7612f9cb, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:48:03,089 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-13) [33e46792] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:48:03,090 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-13) [1e5abacb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:48:03,091 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-13) [1e5abacb] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-98) [48e5821f] Autorecovering 1 hosts >2014-04-30 06:50:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-98) [48e5821f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:50:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 72cadd9 >2014-04-30 06:50:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] FINISH, SetVdsStatusVDSCommand, log id: 72cadd9 >2014-04-30 06:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1fe06961 >2014-04-30 06:50:00,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 54637274 >2014-04-30 06:50:00,727 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] Failed in GetHardwareInfoVDS method >2014-04-30 06:50:00,727 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:50:00,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2e8f26db >2014-04-30 06:50:00,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] HostName = 10.34.27.209 >2014-04-30 06:50:00,727 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:50:00,727 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [cf439cd] FINISH, GetHardwareInfoVDSCommand, log id: 54637274 >2014-04-30 06:50:00,732 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [cf439cd] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:50:00,743 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-98) [3fbeaecf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:50:00,746 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:50:00,748 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1fe06961 >2014-04-30 06:50:00,771 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:50:00,772 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:50:03,454 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [30770c8d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 19ca7bbc >2014-04-30 06:50:03,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [30770c8d] Failed in GetHardwareInfoVDS method >2014-04-30 06:50:03,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [30770c8d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:50:03,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [30770c8d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1dee91ef >2014-04-30 06:50:03,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [30770c8d] HostName = 10.34.27.209 >2014-04-30 06:50:03,687 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [30770c8d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:50:03,687 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-86) [30770c8d] FINISH, GetHardwareInfoVDSCommand, log id: 19ca7bbc >2014-04-30 06:50:03,729 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [30770c8d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:50:03,740 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [30770c8d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:50:03,972 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-86) [3bc22c7d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:50:03,974 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [6fc391d3] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:50:03,980 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [6fc391d3] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 14bd305e >2014-04-30 06:53:03,981 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [6fc391d3] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:53:03,981 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-86) [6fc391d3] FINISH, ConnectStorageServerVDSCommand, log id: 14bd305e >2014-04-30 06:53:03,981 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-86) [6fc391d3] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:53:03,986 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4072e034 >2014-04-30 06:53:04,459 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4072e034 >2014-04-30 06:53:04,459 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:53:04,469 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-86) [379ba16e] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:53:04,470 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [379ba16e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1c605ad7 >2014-04-30 06:53:04,473 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-86) [379ba16e] FINISH, SetVdsStatusVDSCommand, log id: 1c605ad7 >2014-04-30 06:53:04,479 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [379ba16e] Correlation ID: 379ba16e, Job ID: 10cea0c6-65fb-4351-9d73-2f74ea296c1f, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:53:04,487 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [379ba16e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:53:04,489 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-86) [379ba16e] Correlation ID: 3bc22c7d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:53:04,494 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-86) [7dbf4e31] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:53:04,495 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-86) [355eabf0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:53:04,495 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-86) [355eabf0] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 06:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-96) [173d021a] Autorecovering 1 hosts >2014-04-30 06:55:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-96) [173d021a] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 06:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:55:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 06:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 06:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5109772f >2014-04-30 06:55:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] FINISH, SetVdsStatusVDSCommand, log id: 5109772f >2014-04-30 06:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 49038cb >2014-04-30 06:55:00,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 158c4480 >2014-04-30 06:55:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] Failed in GetHardwareInfoVDS method >2014-04-30 06:55:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:55:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2be53a8d >2014-04-30 06:55:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] HostName = 10.34.27.209 >2014-04-30 06:55:00,708 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:55:00,708 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-96) [1cb6bea] FINISH, GetHardwareInfoVDSCommand, log id: 158c4480 >2014-04-30 06:55:00,713 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-96) [1cb6bea] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:55:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-96) [3543ad8] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:55:00,723 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-96) [ad2a786] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:55:00,724 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-96) [ad2a786] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 49038cb >2014-04-30 06:55:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [ad2a786] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 06:55:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-96) [ad2a786] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 06:55:01,861 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [5420242e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 42f8469f >2014-04-30 06:55:02,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [5420242e] Failed in GetHardwareInfoVDS method >2014-04-30 06:55:02,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [5420242e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:55:02,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [5420242e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4cfd1419 >2014-04-30 06:55:02,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [5420242e] HostName = 10.34.27.209 >2014-04-30 06:55:02,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [5420242e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 06:55:02,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-10) [5420242e] FINISH, GetHardwareInfoVDSCommand, log id: 42f8469f >2014-04-30 06:55:02,093 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [5420242e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 06:55:02,104 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [5420242e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 06:55:02,331 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-10) [11e0d16e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:55:02,332 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-10) [75d9095c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 06:55:02,338 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-10) [75d9095c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 784096a >2014-04-30 06:58:02,340 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-10) [75d9095c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 06:58:02,340 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-10) [75d9095c] FINISH, ConnectStorageServerVDSCommand, log id: 784096a >2014-04-30 06:58:02,340 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-10) [75d9095c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 06:58:02,344 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 508cbdc1 >2014-04-30 06:58:02,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 508cbdc1 >2014-04-30 06:58:02,793 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 06:58:02,809 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-10) [5862e251] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:58:02,810 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [5862e251] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 42e402bc >2014-04-30 06:58:02,813 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-10) [5862e251] FINISH, SetVdsStatusVDSCommand, log id: 42e402bc >2014-04-30 06:58:02,821 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [5862e251] Correlation ID: 5862e251, Job ID: 4bb7a99e-1ec6-4d52-92c2-59a2c425de35, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 06:58:02,832 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [5862e251] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 06:58:02,835 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-10) [5862e251] Correlation ID: 11e0d16e, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 06:58:02,840 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-10) [7aa55e3e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:58:02,842 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-10) [45a2c65f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 06:58:02,843 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-10) [45a2c65f] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-11) [398f8810] Autorecovering 1 hosts >2014-04-30 07:00:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-11) [398f8810] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:00:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:00:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 597b3b5 >2014-04-30 07:00:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] FINISH, SetVdsStatusVDSCommand, log id: 597b3b5 >2014-04-30 07:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 559e1511 >2014-04-30 07:00:00,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 63aa3b65 >2014-04-30 07:00:00,720 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] Failed in GetHardwareInfoVDS method >2014-04-30 07:00:00,721 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:00:00,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3d219701 >2014-04-30 07:00:00,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] HostName = 10.34.27.209 >2014-04-30 07:00:00,721 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:00:00,721 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-11) [42e7cc92] FINISH, GetHardwareInfoVDSCommand, log id: 63aa3b65 >2014-04-30 07:00:00,726 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-11) [42e7cc92] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:00:00,736 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-11) [469e7119] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:00:00,737 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-11) [445c681b] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:00:00,738 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-11) [445c681b] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 559e1511 >2014-04-30 07:00:00,753 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [445c681b] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:00:00,754 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-11) [445c681b] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:00:03,203 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 12aa565 >2014-04-30 07:00:03,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] Failed in GetHardwareInfoVDS method >2014-04-30 07:00:03,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:00:03,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@16d08a7c >2014-04-30 07:00:03,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] HostName = 10.34.27.209 >2014-04-30 07:00:03,429 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:00:03,429 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [4950d11d] FINISH, GetHardwareInfoVDSCommand, log id: 12aa565 >2014-04-30 07:00:03,435 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [4950d11d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:00:03,443 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [4950d11d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:00:03,672 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-43) [56b71bdb] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:00:03,673 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [6d4e5402] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:00:03,680 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [6d4e5402] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3f522ebc >2014-04-30 07:03:03,681 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [6d4e5402] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:03:03,681 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-43) [6d4e5402] FINISH, ConnectStorageServerVDSCommand, log id: 3f522ebc >2014-04-30 07:03:03,681 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-43) [6d4e5402] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:03:03,685 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 52fef6ba >2014-04-30 07:03:04,134 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 52fef6ba >2014-04-30 07:03:04,134 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:03:04,248 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-43) [4cf81ebc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:03:04,249 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [4cf81ebc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 260be2fa >2014-04-30 07:03:04,276 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [4cf81ebc] FINISH, SetVdsStatusVDSCommand, log id: 260be2fa >2014-04-30 07:03:04,307 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [4cf81ebc] Correlation ID: 4cf81ebc, Job ID: 7ebdb2d4-4a91-470a-9621-8a5964fb9e61, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:03:04,342 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [4cf81ebc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:03:04,397 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [4cf81ebc] Correlation ID: 56b71bdb, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:03:04,459 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [790b2d54] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:03:04,461 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [2ddda5a1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:03:04,461 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-43) [2ddda5a1] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-5) [3e608cce] Autorecovering 1 hosts >2014-04-30 07:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-5) [3e608cce] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 639c32f >2014-04-30 07:05:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] FINISH, SetVdsStatusVDSCommand, log id: 639c32f >2014-04-30 07:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2f5a1ed0 >2014-04-30 07:05:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 27d6a73c >2014-04-30 07:05:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] Failed in GetHardwareInfoVDS method >2014-04-30 07:05:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:05:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6e928b07 >2014-04-30 07:05:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] HostName = 10.34.27.209 >2014-04-30 07:05:00,710 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:05:00,710 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [150f70cb] FINISH, GetHardwareInfoVDSCommand, log id: 27d6a73c >2014-04-30 07:05:00,715 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [150f70cb] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:05:00,723 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-5) [6ae2f4f5] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:05:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:05:00,725 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2f5a1ed0 >2014-04-30 07:05:00,740 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:05:00,741 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:05:01,863 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5dd7915] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5a4ae5da >2014-04-30 07:05:02,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5dd7915] Failed in GetHardwareInfoVDS method >2014-04-30 07:05:02,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5dd7915] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:05:02,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5dd7915] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@48b241c1 >2014-04-30 07:05:02,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5dd7915] HostName = 10.34.27.209 >2014-04-30 07:05:02,089 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5dd7915] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:05:02,089 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [5dd7915] FINISH, GetHardwareInfoVDSCommand, log id: 5a4ae5da >2014-04-30 07:05:02,145 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [5dd7915] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:05:02,156 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [5dd7915] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:05:02,382 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-46) [2bddbb41] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:05:02,384 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-46) [3149593c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:05:02,390 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [3149593c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 34f61fa4 >2014-04-30 07:08:02,391 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [3149593c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:08:02,391 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [3149593c] FINISH, ConnectStorageServerVDSCommand, log id: 34f61fa4 >2014-04-30 07:08:02,391 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-46) [3149593c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:08:02,397 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 499caff7 >2014-04-30 07:08:02,855 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 499caff7 >2014-04-30 07:08:02,855 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:08:02,866 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-46) [6d5a90d1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:08:02,867 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [6d5a90d1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 412a691c >2014-04-30 07:08:02,870 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [6d5a90d1] FINISH, SetVdsStatusVDSCommand, log id: 412a691c >2014-04-30 07:08:02,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [6d5a90d1] Correlation ID: 6d5a90d1, Job ID: 66bb7223-a783-4f60-9e88-be96dd965f2b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:08:02,887 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [6d5a90d1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:08:02,889 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [6d5a90d1] Correlation ID: 2bddbb41, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:08:02,893 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-46) [51f63819] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:08:02,894 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-46) [764adf7a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:08:02,895 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) [764adf7a] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:10:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-60) [4badcb49] Autorecovering 1 hosts >2014-04-30 07:10:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-60) [4badcb49] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:10:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:10:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:10:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:10:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 14b2557 >2014-04-30 07:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] FINISH, SetVdsStatusVDSCommand, log id: 14b2557 >2014-04-30 07:10:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6e211c34 >2014-04-30 07:10:00,496 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5c12f198 >2014-04-30 07:10:00,722 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] Failed in GetHardwareInfoVDS method >2014-04-30 07:10:00,722 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:10:00,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5fb36161 >2014-04-30 07:10:00,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] HostName = 10.34.27.209 >2014-04-30 07:10:00,722 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:10:00,722 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [29de148c] FINISH, GetHardwareInfoVDSCommand, log id: 5c12f198 >2014-04-30 07:10:00,739 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [29de148c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:10:00,749 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-60) [5583b000] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:10:00,751 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-60) [4b138017] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:10:00,751 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) [4b138017] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6e211c34 >2014-04-30 07:10:00,764 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [4b138017] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:10:00,765 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [4b138017] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:10:03,263 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 57af5fe >2014-04-30 07:10:03,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] Failed in GetHardwareInfoVDS method >2014-04-30 07:10:03,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:10:03,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@26946bb4 >2014-04-30 07:10:03,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] HostName = 10.34.27.209 >2014-04-30 07:10:03,490 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:10:03,490 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-76) [52052a18] FINISH, GetHardwareInfoVDSCommand, log id: 57af5fe >2014-04-30 07:10:03,494 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [52052a18] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:10:03,504 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [52052a18] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:10:03,729 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-76) [2e259768] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:10:03,730 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-76) [30e733d0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:10:03,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-76) [30e733d0] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 614f719a >2014-04-30 07:13:03,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-76) [30e733d0] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:13:03,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-76) [30e733d0] FINISH, ConnectStorageServerVDSCommand, log id: 614f719a >2014-04-30 07:13:03,738 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-76) [30e733d0] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:13:03,741 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 42bbdd42 >2014-04-30 07:13:04,189 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 42bbdd42 >2014-04-30 07:13:04,189 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:13:04,203 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-76) [39f394a8] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:13:04,204 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [39f394a8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 632690ec >2014-04-30 07:13:04,206 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-76) [39f394a8] FINISH, SetVdsStatusVDSCommand, log id: 632690ec >2014-04-30 07:13:04,212 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [39f394a8] Correlation ID: 39f394a8, Job ID: 6949959f-52c8-47c4-9b1d-61086dc5eb60, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:13:04,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [39f394a8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:13:04,221 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-76) [39f394a8] Correlation ID: 2e259768, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:13:04,227 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-76) [2b78b501] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:13:04,228 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-76) [11eeed00] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:13:04,229 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-76) [11eeed00] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-88) [6aba72d1] Autorecovering 1 hosts >2014-04-30 07:15:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-88) [6aba72d1] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:15:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:15:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1901351f >2014-04-30 07:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] FINISH, SetVdsStatusVDSCommand, log id: 1901351f >2014-04-30 07:15:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 126ca019 >2014-04-30 07:15:00,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 669d49e1 >2014-04-30 07:15:00,711 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] Failed in GetHardwareInfoVDS method >2014-04-30 07:15:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:15:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@605d8768 >2014-04-30 07:15:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] HostName = 10.34.27.209 >2014-04-30 07:15:00,712 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:15:00,712 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-88) [9754ae1] FINISH, GetHardwareInfoVDSCommand, log id: 669d49e1 >2014-04-30 07:15:00,721 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-88) [9754ae1] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:15:00,737 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-88) [dc28b1e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:15:00,738 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-88) [ef5d85d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:15:00,739 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-88) [ef5d85d] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 126ca019 >2014-04-30 07:15:00,757 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [ef5d85d] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:15:00,758 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-88) [ef5d85d] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:15:01,587 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [28f8ef11] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 21259821 >2014-04-30 07:15:01,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [28f8ef11] Failed in GetHardwareInfoVDS method >2014-04-30 07:15:01,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [28f8ef11] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:15:01,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [28f8ef11] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@78cdde32 >2014-04-30 07:15:01,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [28f8ef11] HostName = 10.34.27.209 >2014-04-30 07:15:01,813 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [28f8ef11] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:15:01,813 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [28f8ef11] FINISH, GetHardwareInfoVDSCommand, log id: 21259821 >2014-04-30 07:15:01,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [28f8ef11] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:15:01,830 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [28f8ef11] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:15:02,055 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-7) [35aaa97e] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:15:02,057 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [37ff3ad0] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:15:02,063 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [37ff3ad0] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6646fa22 >2014-04-30 07:18:02,065 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [37ff3ad0] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:18:02,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-7) [37ff3ad0] FINISH, ConnectStorageServerVDSCommand, log id: 6646fa22 >2014-04-30 07:18:02,065 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-7) [37ff3ad0] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:18:02,069 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5663cf61 >2014-04-30 07:18:02,530 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 5663cf61 >2014-04-30 07:18:02,531 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:18:02,553 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-7) [183435df] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:18:02,554 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [183435df] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 486f9d34 >2014-04-30 07:18:02,557 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [183435df] FINISH, SetVdsStatusVDSCommand, log id: 486f9d34 >2014-04-30 07:18:02,563 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [183435df] Correlation ID: 183435df, Job ID: 678047ac-346e-4eb8-8dad-13366a17edbd, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:18:02,571 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [183435df] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:18:02,573 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [183435df] Correlation ID: 35aaa97e, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:18:02,577 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-7) [11e9d16e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:18:02,578 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [6d2acd47] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:18:02,579 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-7) [6d2acd47] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-4) [50fdeab5] Autorecovering 1 hosts >2014-04-30 07:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-4) [50fdeab5] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:20:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 10b12fc5 >2014-04-30 07:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] FINISH, SetVdsStatusVDSCommand, log id: 10b12fc5 >2014-04-30 07:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6d8903f >2014-04-30 07:20:00,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 1d452a02 >2014-04-30 07:20:00,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] Failed in GetHardwareInfoVDS method >2014-04-30 07:20:00,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:20:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@27c8f851 >2014-04-30 07:20:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] HostName = 10.34.27.209 >2014-04-30 07:20:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:20:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [4bf35828] FINISH, GetHardwareInfoVDSCommand, log id: 1d452a02 >2014-04-30 07:20:00,698 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [4bf35828] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:20:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-4) [795d4c54] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:20:00,711 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:20:00,712 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6d8903f >2014-04-30 07:20:00,729 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:20:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:20:02,935 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [59b47c84] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6a6cb011 >2014-04-30 07:20:03,154 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [59b47c84] Failed in GetHardwareInfoVDS method >2014-04-30 07:20:03,154 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [59b47c84] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:20:03,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [59b47c84] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@677bbeeb >2014-04-30 07:20:03,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [59b47c84] HostName = 10.34.27.209 >2014-04-30 07:20:03,154 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [59b47c84] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:20:03,154 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-69) [59b47c84] FINISH, GetHardwareInfoVDSCommand, log id: 6a6cb011 >2014-04-30 07:20:03,172 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [59b47c84] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:20:03,195 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [59b47c84] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:20:03,422 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-69) [2702c17b] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:20:03,425 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [1ce24c7e] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:20:03,436 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [1ce24c7e] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 66fa5fed >2014-04-30 07:23:03,438 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [1ce24c7e] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:23:03,438 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-69) [1ce24c7e] FINISH, ConnectStorageServerVDSCommand, log id: 66fa5fed >2014-04-30 07:23:03,438 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-69) [1ce24c7e] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:23:03,443 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 123dbd67 >2014-04-30 07:23:03,888 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 123dbd67 >2014-04-30 07:23:03,888 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:23:03,899 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-69) [2b32adc2] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:23:03,900 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [2b32adc2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 31d2b4fb >2014-04-30 07:23:03,903 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-69) [2b32adc2] FINISH, SetVdsStatusVDSCommand, log id: 31d2b4fb >2014-04-30 07:23:03,908 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [2b32adc2] Correlation ID: 2b32adc2, Job ID: d7c6d9ea-8de3-45bf-ac58-d7bd092c201c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:23:03,915 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [2b32adc2] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:23:03,917 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-69) [2b32adc2] Correlation ID: 2702c17b, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:23:03,922 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-69) [7a18a02f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:23:03,923 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-69) [4ec756b8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:23:03,924 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-69) [4ec756b8] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) [1d5f1b42] Autorecovering 1 hosts >2014-04-30 07:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) [1d5f1b42] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:25:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:25:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4d66688 >2014-04-30 07:25:00,017 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] FINISH, SetVdsStatusVDSCommand, log id: 4d66688 >2014-04-30 07:25:00,019 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 39bc7d17 >2014-04-30 07:25:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6f4e46c4 >2014-04-30 07:25:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] Failed in GetHardwareInfoVDS method >2014-04-30 07:25:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:25:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7c0826f2 >2014-04-30 07:25:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] HostName = 10.34.27.209 >2014-04-30 07:25:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:25:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e36084d] FINISH, GetHardwareInfoVDSCommand, log id: 6f4e46c4 >2014-04-30 07:25:00,701 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [2e36084d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:25:00,711 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-54) [7165852c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:25:00,712 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-54) [31044de5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:25:00,713 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) [31044de5] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 39bc7d17 >2014-04-30 07:25:00,728 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [31044de5] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:25:00,729 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [31044de5] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:25:01,274 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3b213f62] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4fce1374 >2014-04-30 07:25:01,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3b213f62] Failed in GetHardwareInfoVDS method >2014-04-30 07:25:01,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3b213f62] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:25:01,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3b213f62] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4643b7fb >2014-04-30 07:25:01,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3b213f62] HostName = 10.34.27.209 >2014-04-30 07:25:01,493 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3b213f62] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:25:01,493 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-26) [3b213f62] FINISH, GetHardwareInfoVDSCommand, log id: 4fce1374 >2014-04-30 07:25:01,497 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [3b213f62] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:25:01,507 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [3b213f62] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:25:01,728 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-26) [654c61d4] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:25:01,730 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [3c9e1d7d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:25:01,736 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [3c9e1d7d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6bc17ea2 >2014-04-30 07:28:01,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [3c9e1d7d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:28:01,738 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-26) [3c9e1d7d] FINISH, ConnectStorageServerVDSCommand, log id: 6bc17ea2 >2014-04-30 07:28:01,738 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-26) [3c9e1d7d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:28:01,742 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 106477aa >2014-04-30 07:28:02,186 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: 106477aa >2014-04-30 07:28:02,186 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:28:02,208 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-26) [cce4fbd] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:28:02,209 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [cce4fbd] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4be45100 >2014-04-30 07:28:02,211 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-26) [cce4fbd] FINISH, SetVdsStatusVDSCommand, log id: 4be45100 >2014-04-30 07:28:02,218 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [cce4fbd] Correlation ID: cce4fbd, Job ID: ba62ed6a-6995-4dcd-b736-0160c6963396, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:28:02,224 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [cce4fbd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:28:02,226 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-26) [cce4fbd] Correlation ID: 654c61d4, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:28:02,230 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-26) [62c7848d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:28:02,231 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-26) [65f17ab6] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:28:02,231 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-26) [65f17ab6] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-49) [34373ba8] Autorecovering 1 hosts >2014-04-30 07:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-49) [34373ba8] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:30:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 40d46d95 >2014-04-30 07:30:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] FINISH, SetVdsStatusVDSCommand, log id: 40d46d95 >2014-04-30 07:30:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4a7da0c >2014-04-30 07:30:01,141 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 72b98c4d >2014-04-30 07:30:01,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] Failed in GetHardwareInfoVDS method >2014-04-30 07:30:01,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:30:01,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@73aa4e12 >2014-04-30 07:30:01,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] HostName = 10.34.27.209 >2014-04-30 07:30:01,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:30:01,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5fdcd707] FINISH, GetHardwareInfoVDSCommand, log id: 72b98c4d >2014-04-30 07:30:01,700 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [5fdcd707] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:30:01,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-49) [5605ce7d] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:30:01,709 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-49) [5a770a75] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:30:01,710 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [5a770a75] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4a7da0c >2014-04-30 07:30:01,724 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5a770a75] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:30:01,725 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5a770a75] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:30:02,939 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3bb2d659 >2014-04-30 07:30:03,507 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] Failed in GetHardwareInfoVDS method >2014-04-30 07:30:03,507 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:30:03,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@298904d4 >2014-04-30 07:30:03,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] HostName = 10.34.27.209 >2014-04-30 07:30:03,507 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:30:03,507 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-36) [17e66f10] FINISH, GetHardwareInfoVDSCommand, log id: 3bb2d659 >2014-04-30 07:30:03,514 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [17e66f10] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:30:03,523 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [17e66f10] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:30:04,107 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-36) [57a0ef91] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:30:04,109 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [4ca0e9a6] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:30:04,116 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [4ca0e9a6] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3004a8e5 >2014-04-30 07:33:04,118 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [4ca0e9a6] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:33:04,118 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-36) [4ca0e9a6] FINISH, ConnectStorageServerVDSCommand, log id: 3004a8e5 >2014-04-30 07:33:04,118 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-36) [4ca0e9a6] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:33:04,122 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 70132706 >2014-04-30 07:33:04,557 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 70132706 >2014-04-30 07:33:04,557 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:33:04,567 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-36) [7afb9b4c] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:33:04,568 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [7afb9b4c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 272127bd >2014-04-30 07:33:04,571 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-36) [7afb9b4c] FINISH, SetVdsStatusVDSCommand, log id: 272127bd >2014-04-30 07:33:04,577 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [7afb9b4c] Correlation ID: 7afb9b4c, Job ID: f7a9eb69-eb9e-4d3b-91a6-0d531a5dc591, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:33:04,586 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [7afb9b4c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:33:04,588 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-36) [7afb9b4c] Correlation ID: 57a0ef91, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:33:04,592 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-36) [3abf68e6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:33:04,594 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-36) [7de1d183] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:33:04,594 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-36) [7de1d183] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [6d2acd47] Autorecovering 1 hosts >2014-04-30 07:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-7) [6d2acd47] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:35:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 785fa4d9 >2014-04-30 07:35:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] FINISH, SetVdsStatusVDSCommand, log id: 785fa4d9 >2014-04-30 07:35:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1ddf231e >2014-04-30 07:35:00,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 332359e2 >2014-04-30 07:35:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] Failed in GetHardwareInfoVDS method >2014-04-30 07:35:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:35:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@55e21ea8 >2014-04-30 07:35:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] HostName = 10.34.27.209 >2014-04-30 07:35:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:35:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-7) [c2cb076] FINISH, GetHardwareInfoVDSCommand, log id: 332359e2 >2014-04-30 07:35:00,697 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-7) [c2cb076] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:35:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-7) [3bb88175] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:35:00,717 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-7) [1773cc6f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:35:00,717 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-7) [1773cc6f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1ddf231e >2014-04-30 07:35:00,734 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [1773cc6f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:35:00,735 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-7) [1773cc6f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:35:01,944 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: b1de60f >2014-04-30 07:35:02,173 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] Failed in GetHardwareInfoVDS method >2014-04-30 07:35:02,173 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:35:02,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@57eae87c >2014-04-30 07:35:02,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] HostName = 10.34.27.209 >2014-04-30 07:35:02,173 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:35:02,173 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-32) [407b01d9] FINISH, GetHardwareInfoVDSCommand, log id: b1de60f >2014-04-30 07:35:02,178 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [407b01d9] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:35:02,187 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [407b01d9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:35:02,406 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-32) [1a63a843] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:35:02,408 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-32) [67f7f3c2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:35:02,415 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [67f7f3c2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5cd91727 >2014-04-30 07:36:05,026 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-9) [4d79f29e] Start refreshing all users data >2014-04-30 07:36:07,315 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [19235d27] Attempting to update VMs/Templates Ovf. >2014-04-30 07:36:07,315 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [19235d27] Attempting to update VM OVFs in Data Center NFS >2014-04-30 07:36:07,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [19235d27] Successfully updated VM OVFs in Data Center NFS >2014-04-30 07:36:07,316 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [19235d27] Attempting to update template OVFs in Data Center NFS >2014-04-30 07:36:07,317 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [19235d27] Successfully updated templates OVFs in Data Center NFS >2014-04-30 07:36:07,317 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [19235d27] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-30 07:36:07,317 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-42) [19235d27] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-30 07:38:02,416 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [67f7f3c2] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:38:02,416 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-32) [67f7f3c2] FINISH, ConnectStorageServerVDSCommand, log id: 5cd91727 >2014-04-30 07:38:02,416 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-32) [67f7f3c2] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:38:02,420 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4a9eb371 >2014-04-30 07:38:02,853 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 4a9eb371 >2014-04-30 07:38:02,853 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:38:02,867 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-32) [2827c33a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:38:02,868 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [2827c33a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 5dc7e632 >2014-04-30 07:38:02,870 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-32) [2827c33a] FINISH, SetVdsStatusVDSCommand, log id: 5dc7e632 >2014-04-30 07:38:02,876 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [2827c33a] Correlation ID: 2827c33a, Job ID: a052f5fc-c38d-4808-88a2-d5f0a3604ec3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:38:02,884 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [2827c33a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:38:02,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-32) [2827c33a] Correlation ID: 1a63a843, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:38:02,890 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-32) [5e8ff9d3] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:38:02,891 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-32) [5ee2de65] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:38:02,892 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-32) [5ee2de65] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-62) [39584f9] Autorecovering 1 hosts >2014-04-30 07:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-62) [39584f9] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:40:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1262a180 >2014-04-30 07:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] FINISH, SetVdsStatusVDSCommand, log id: 1262a180 >2014-04-30 07:40:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 195fd4a2 >2014-04-30 07:40:00,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2bcbd8c4 >2014-04-30 07:40:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] Failed in GetHardwareInfoVDS method >2014-04-30 07:40:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:40:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@54044252 >2014-04-30 07:40:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] HostName = 10.34.27.209 >2014-04-30 07:40:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:40:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [3fa2372] FINISH, GetHardwareInfoVDSCommand, log id: 2bcbd8c4 >2014-04-30 07:40:00,694 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [3fa2372] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:40:00,704 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-62) [17f3e34f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:40:00,705 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-62) [482631c] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:40:00,706 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-62) [482631c] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 195fd4a2 >2014-04-30 07:40:00,721 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [482631c] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:40:00,722 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-62) [482631c] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:40:03,275 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [5c65b29f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7f5dd597 >2014-04-30 07:40:03,495 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [5c65b29f] Failed in GetHardwareInfoVDS method >2014-04-30 07:40:03,495 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [5c65b29f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:40:03,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [5c65b29f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6971edd4 >2014-04-30 07:40:03,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [5c65b29f] HostName = 10.34.27.209 >2014-04-30 07:40:03,495 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [5c65b29f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:40:03,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-29) [5c65b29f] FINISH, GetHardwareInfoVDSCommand, log id: 7f5dd597 >2014-04-30 07:40:03,500 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [5c65b29f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:40:03,511 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [5c65b29f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:40:03,730 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-29) [7a86f2d1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:40:03,735 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [666573bb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:40:03,744 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [666573bb] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 52d44244 >2014-04-30 07:43:03,745 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [666573bb] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:43:03,745 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-29) [666573bb] FINISH, ConnectStorageServerVDSCommand, log id: 52d44244 >2014-04-30 07:43:03,745 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-29) [666573bb] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:43:03,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 354f7bab >2014-04-30 07:43:04,182 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 354f7bab >2014-04-30 07:43:04,182 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:43:04,193 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-29) [3f30b98d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:43:04,194 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [3f30b98d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 526fd93e >2014-04-30 07:43:04,197 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-29) [3f30b98d] FINISH, SetVdsStatusVDSCommand, log id: 526fd93e >2014-04-30 07:43:04,203 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [3f30b98d] Correlation ID: 3f30b98d, Job ID: 12a60f8c-2af4-430c-a81d-8a8f17348035, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:43:04,213 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [3f30b98d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:43:04,216 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-29) [3f30b98d] Correlation ID: 7a86f2d1, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:43:04,220 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-29) [e01598b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:43:04,221 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-29) [301ea09e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:43:04,221 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-29) [301ea09e] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-47) [63374716] Autorecovering 1 hosts >2014-04-30 07:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-47) [63374716] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:45:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:45:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 10ef66bf >2014-04-30 07:45:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] FINISH, SetVdsStatusVDSCommand, log id: 10ef66bf >2014-04-30 07:45:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: cd2846 >2014-04-30 07:45:00,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7e5fe53c >2014-04-30 07:45:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] Failed in GetHardwareInfoVDS method >2014-04-30 07:45:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:45:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4d605423 >2014-04-30 07:45:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] HostName = 10.34.27.209 >2014-04-30 07:45:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:45:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-47) [6bc5b380] FINISH, GetHardwareInfoVDSCommand, log id: 7e5fe53c >2014-04-30 07:45:00,696 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-47) [6bc5b380] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:45:00,705 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-47) [40d70a01] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:45:00,706 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-47) [2231d6f0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:45:00,707 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-47) [2231d6f0] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: cd2846 >2014-04-30 07:45:00,721 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [2231d6f0] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:45:00,722 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-47) [2231d6f0] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:45:01,573 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f5a836] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 402bad1e >2014-04-30 07:45:01,792 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f5a836] Failed in GetHardwareInfoVDS method >2014-04-30 07:45:01,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f5a836] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:45:01,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f5a836] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7372ad7b >2014-04-30 07:45:01,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f5a836] HostName = 10.34.27.209 >2014-04-30 07:45:01,793 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f5a836] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:45:01,793 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [4f5a836] FINISH, GetHardwareInfoVDSCommand, log id: 402bad1e >2014-04-30 07:45:01,798 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [4f5a836] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:45:01,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [4f5a836] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:45:02,028 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-93) [2abfb648] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:45:02,030 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [2a48947d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:45:02,036 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [2a48947d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3cc0460f >2014-04-30 07:48:02,038 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [2a48947d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:48:02,038 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-93) [2a48947d] FINISH, ConnectStorageServerVDSCommand, log id: 3cc0460f >2014-04-30 07:48:02,038 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-93) [2a48947d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:48:02,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4cc55583 >2014-04-30 07:48:02,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 4cc55583 >2014-04-30 07:48:02,489 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:48:02,502 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-93) [7032c77] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:48:02,503 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [7032c77] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7fb70c64 >2014-04-30 07:48:02,506 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [7032c77] FINISH, SetVdsStatusVDSCommand, log id: 7fb70c64 >2014-04-30 07:48:02,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [7032c77] Correlation ID: 7032c77, Job ID: 55440f3f-2ac8-48d6-93fd-d2167031a854, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:48:02,523 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [7032c77] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:48:02,528 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [7032c77] Correlation ID: 2abfb648, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:48:02,534 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [3b0d1ac9] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:48:02,535 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [629a4741] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:48:02,536 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-93) [629a4741] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:50:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) [31044de5] Autorecovering 1 hosts >2014-04-30 07:50:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) [31044de5] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:50:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:50:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:50:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:50:00,008 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 5693090e >2014-04-30 07:50:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] FINISH, SetVdsStatusVDSCommand, log id: 5693090e >2014-04-30 07:50:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 764192ed >2014-04-30 07:50:00,473 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 50e3ef36 >2014-04-30 07:50:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] Failed in GetHardwareInfoVDS method >2014-04-30 07:50:00,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:50:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2c704a6c >2014-04-30 07:50:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] HostName = 10.34.27.209 >2014-04-30 07:50:00,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:50:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] FINISH, GetHardwareInfoVDSCommand, log id: 50e3ef36 >2014-04-30 07:50:00,697 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [4f9bec9f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:50:00,704 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-54) [2e13029e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:50:00,705 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-54) [2e159e5e] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:50:00,706 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) [2e159e5e] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 764192ed >2014-04-30 07:50:00,738 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [2e159e5e] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:50:00,740 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [2e159e5e] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:50:02,897 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 605a06f1 >2014-04-30 07:50:03,117 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] Failed in GetHardwareInfoVDS method >2014-04-30 07:50:03,117 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:50:03,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@367de9a2 >2014-04-30 07:50:03,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] HostName = 10.34.27.209 >2014-04-30 07:50:03,117 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:50:03,117 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-16) [4a50c78f] FINISH, GetHardwareInfoVDSCommand, log id: 605a06f1 >2014-04-30 07:50:03,131 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [4a50c78f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:50:03,155 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [4a50c78f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:50:03,374 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-16) [5c9765d2] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:50:03,376 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-16) [7ad8334] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:50:03,382 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-16) [7ad8334] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 45ba6d30 >2014-04-30 07:53:03,383 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-16) [7ad8334] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:53:03,383 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-16) [7ad8334] FINISH, ConnectStorageServerVDSCommand, log id: 45ba6d30 >2014-04-30 07:53:03,384 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-16) [7ad8334] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:53:03,387 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 5b3c3d25 >2014-04-30 07:53:03,846 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 5b3c3d25 >2014-04-30 07:53:03,846 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:53:03,857 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-16) [11c11ba1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:53:03,858 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [11c11ba1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 558df771 >2014-04-30 07:53:03,861 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-16) [11c11ba1] FINISH, SetVdsStatusVDSCommand, log id: 558df771 >2014-04-30 07:53:03,868 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [11c11ba1] Correlation ID: 11c11ba1, Job ID: 4bef4489-62fb-442d-ad36-f04b312cb95f, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:53:03,877 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [11c11ba1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:53:03,879 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-16) [11c11ba1] Correlation ID: 5c9765d2, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:53:03,883 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-16) [52d8d5e6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:53:03,884 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-16) [3f374f26] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:53:03,885 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-16) [3f374f26] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 07:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-64) [1652f88e] Autorecovering 1 hosts >2014-04-30 07:55:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-64) [1652f88e] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 07:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 07:55:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 07:55:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7fb5b663 >2014-04-30 07:55:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] FINISH, SetVdsStatusVDSCommand, log id: 7fb5b663 >2014-04-30 07:55:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 144bfc74 >2014-04-30 07:55:00,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 75d805e >2014-04-30 07:55:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] Failed in GetHardwareInfoVDS method >2014-04-30 07:55:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:55:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4a472733 >2014-04-30 07:55:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] HostName = 10.34.27.209 >2014-04-30 07:55:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:55:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [b97c315] FINISH, GetHardwareInfoVDSCommand, log id: 75d805e >2014-04-30 07:55:00,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [b97c315] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:55:00,702 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [3c90d974] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:55:00,703 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:55:00,704 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 144bfc74 >2014-04-30 07:55:00,750 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 07:55:00,751 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 07:55:01,234 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 690e1a82 >2014-04-30 07:55:01,456 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] Failed in GetHardwareInfoVDS method >2014-04-30 07:55:01,456 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:55:01,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@1cee82b0 >2014-04-30 07:55:01,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] HostName = 10.34.27.209 >2014-04-30 07:55:01,456 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 07:55:01,456 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-5) [10f6b248] FINISH, GetHardwareInfoVDSCommand, log id: 690e1a82 >2014-04-30 07:55:01,461 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [10f6b248] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 07:55:01,470 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [10f6b248] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 07:55:01,689 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-5) [6cd4ede0] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:55:01,691 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-5) [5a19a9eb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 07:55:01,698 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-5) [5a19a9eb] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 6c231790 >2014-04-30 07:58:01,700 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-5) [5a19a9eb] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 07:58:01,700 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-5) [5a19a9eb] FINISH, ConnectStorageServerVDSCommand, log id: 6c231790 >2014-04-30 07:58:01,700 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-5) [5a19a9eb] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 07:58:01,703 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 40b59ac9 >2014-04-30 07:58:02,143 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-48) FINISH, ConnectStoragePoolVDSCommand, log id: 40b59ac9 >2014-04-30 07:58:02,143 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-48) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 07:58:02,154 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-5) [778fa321] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:58:02,155 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [778fa321] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1a5f84aa >2014-04-30 07:58:02,158 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-5) [778fa321] FINISH, SetVdsStatusVDSCommand, log id: 1a5f84aa >2014-04-30 07:58:02,164 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [778fa321] Correlation ID: 778fa321, Job ID: f6325616-1636-4a36-85a6-8b2b7f7c51d5, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 07:58:02,171 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [778fa321] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 07:58:02,173 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-5) [778fa321] Correlation ID: 6cd4ede0, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 07:58:02,177 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-5) [5ced8dc4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:58:02,178 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-5) [2fd18724] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 07:58:02,179 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-5) [2fd18724] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-92) [22bf197f] Autorecovering 1 hosts >2014-04-30 08:00:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-92) [22bf197f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:00:00,004 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:00:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 33eb5727 >2014-04-30 08:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] FINISH, SetVdsStatusVDSCommand, log id: 33eb5727 >2014-04-30 08:00:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 71a92507 >2014-04-30 08:00:00,489 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 32d191b9 >2014-04-30 08:00:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] Failed in GetHardwareInfoVDS method >2014-04-30 08:00:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:00:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5cb622bf >2014-04-30 08:00:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] HostName = 10.34.27.209 >2014-04-30 08:00:00,709 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:00:00,709 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] FINISH, GetHardwareInfoVDSCommand, log id: 32d191b9 >2014-04-30 08:00:00,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-92) [6ba7ee73] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:00:00,723 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-92) [15977de1] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:00:00,724 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-92) [28bd6239] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:00:00,725 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-92) [28bd6239] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 71a92507 >2014-04-30 08:00:00,739 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [28bd6239] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:00:00,740 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-92) [28bd6239] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:00:02,535 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [58a53c0f] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4b61494e >2014-04-30 08:00:02,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [58a53c0f] Failed in GetHardwareInfoVDS method >2014-04-30 08:00:02,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [58a53c0f] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:00:02,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [58a53c0f] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@101e62bf >2014-04-30 08:00:02,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [58a53c0f] HostName = 10.34.27.209 >2014-04-30 08:00:02,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [58a53c0f] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:00:02,783 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-57) [58a53c0f] FINISH, GetHardwareInfoVDSCommand, log id: 4b61494e >2014-04-30 08:00:02,787 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [58a53c0f] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:00:02,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [58a53c0f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:00:03,015 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-57) [14f7f22f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:00:03,017 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [f71f218] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:00:03,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [f71f218] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3addeacd >2014-04-30 08:03:03,025 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [f71f218] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:03:03,025 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-57) [f71f218] FINISH, ConnectStorageServerVDSCommand, log id: 3addeacd >2014-04-30 08:03:03,025 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-57) [f71f218] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:03:03,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1f741a92 >2014-04-30 08:03:04,065 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 1f741a92 >2014-04-30 08:03:04,065 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:03:04,086 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-57) [50c50f3a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:03:04,088 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [50c50f3a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7706b88 >2014-04-30 08:03:04,091 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-57) [50c50f3a] FINISH, SetVdsStatusVDSCommand, log id: 7706b88 >2014-04-30 08:03:04,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [50c50f3a] Correlation ID: 50c50f3a, Job ID: 7f0ddaad-f03e-4070-8f33-95bb32fd91c3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:03:04,107 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [50c50f3a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:03:04,109 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-57) [50c50f3a] Correlation ID: 14f7f22f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:03:04,115 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-57) [60a2a1eb] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:03:04,116 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-57) [5fbef514] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:03:04,117 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-57) [5fbef514] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [780ec1a0] Autorecovering 1 hosts >2014-04-30 08:05:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-79) [780ec1a0] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:05:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 74b043fa >2014-04-30 08:05:00,018 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] FINISH, SetVdsStatusVDSCommand, log id: 74b043fa >2014-04-30 08:05:00,020 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 94dbf28 >2014-04-30 08:05:01,115 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 7e921a1 >2014-04-30 08:05:01,660 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] Failed in GetHardwareInfoVDS method >2014-04-30 08:05:01,660 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:05:01,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@657814c2 >2014-04-30 08:05:01,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] HostName = 10.34.27.209 >2014-04-30 08:05:01,661 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:05:01,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-79) [7b311fd8] FINISH, GetHardwareInfoVDSCommand, log id: 7e921a1 >2014-04-30 08:05:01,666 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-79) [7b311fd8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:05:01,675 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-79) [2d36d5cf] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:05:01,677 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-79) [231c94ae] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:05:01,677 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-79) [231c94ae] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 94dbf28 >2014-04-30 08:05:01,692 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [231c94ae] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:05:01,693 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-79) [231c94ae] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:05:04,837 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2be58ae2 >2014-04-30 08:05:05,405 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] Failed in GetHardwareInfoVDS method >2014-04-30 08:05:05,406 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:05:05,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@e0e0ffa >2014-04-30 08:05:05,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] HostName = 10.34.27.209 >2014-04-30 08:05:05,406 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:05:05,406 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-70) [1e8d0255] FINISH, GetHardwareInfoVDSCommand, log id: 2be58ae2 >2014-04-30 08:05:05,411 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [1e8d0255] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:05:05,423 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [1e8d0255] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:05:06,001 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-70) [5bc1fa8] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:05:06,003 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-70) [3ed6f6fb] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:05:06,009 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [3ed6f6fb] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 630c94a5 >2014-04-30 08:08:06,011 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [3ed6f6fb] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:08:06,011 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-70) [3ed6f6fb] FINISH, ConnectStorageServerVDSCommand, log id: 630c94a5 >2014-04-30 08:08:06,011 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-70) [3ed6f6fb] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:08:06,014 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 4af2de38 >2014-04-30 08:08:11,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 4af2de38 >2014-04-30 08:08:11,483 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:08:11,503 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-70) [43f2787d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:08:11,504 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [43f2787d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 736af47c >2014-04-30 08:08:11,507 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-70) [43f2787d] FINISH, SetVdsStatusVDSCommand, log id: 736af47c >2014-04-30 08:08:11,513 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [43f2787d] Correlation ID: 43f2787d, Job ID: e269f5d4-2510-4878-a821-9e70d29bc4e7, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:08:11,521 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [43f2787d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:08:11,522 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-70) [43f2787d] Correlation ID: 5bc1fa8, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:08:11,527 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-70) [31c82601] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:08:11,528 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-70) [3c286323] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:08:11,528 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-70) [3c286323] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) [429b86b3] Autorecovering 1 hosts >2014-04-30 08:10:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-81) [429b86b3] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [227b194] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [227b194] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [227b194] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:10:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [227b194] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:10:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [227b194] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 10d14780 >2014-04-30 08:10:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-81) [227b194] FINISH, SetVdsStatusVDSCommand, log id: 10d14780 >2014-04-30 08:10:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) [227b194] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 31c89ead >2014-04-30 08:10:00,503 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [227b194] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 472f2eb1 >2014-04-30 08:10:00,738 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [227b194] Failed in GetHardwareInfoVDS method >2014-04-30 08:10:00,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [227b194] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:10:00,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [227b194] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6aa28af9 >2014-04-30 08:10:00,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [227b194] HostName = 10.34.27.209 >2014-04-30 08:10:00,739 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [227b194] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:10:00,739 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-81) [227b194] FINISH, GetHardwareInfoVDSCommand, log id: 472f2eb1 >2014-04-30 08:10:00,753 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-81) [227b194] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:10:00,777 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-81) [31795c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:10:00,778 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-81) [7ff0ea0f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:10:00,779 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-81) [7ff0ea0f] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 31c89ead >2014-04-30 08:10:00,799 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [7ff0ea0f] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:10:00,800 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-81) [7ff0ea0f] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:10:02,902 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 53a9f3d4 >2014-04-30 08:10:03,136 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] Failed in GetHardwareInfoVDS method >2014-04-30 08:10:03,136 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:10:03,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@3ac70e35 >2014-04-30 08:10:03,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] HostName = 10.34.27.209 >2014-04-30 08:10:03,137 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:10:03,137 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-71) [700b1a] FINISH, GetHardwareInfoVDSCommand, log id: 53a9f3d4 >2014-04-30 08:10:03,141 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [700b1a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:10:03,149 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [700b1a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:10:03,384 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-71) [5b63ef7] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:10:03,387 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [6dc9ecd9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:10:03,396 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [6dc9ecd9] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2e1209df >2014-04-30 08:13:03,398 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [6dc9ecd9] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:13:03,398 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-71) [6dc9ecd9] FINISH, ConnectStorageServerVDSCommand, log id: 2e1209df >2014-04-30 08:13:03,398 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-71) [6dc9ecd9] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:13:03,401 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 1c03bc4a >2014-04-30 08:13:03,835 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 1c03bc4a >2014-04-30 08:13:03,835 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:13:03,848 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-71) [124fad3d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:13:03,849 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [124fad3d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 20799654 >2014-04-30 08:13:03,851 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-71) [124fad3d] FINISH, SetVdsStatusVDSCommand, log id: 20799654 >2014-04-30 08:13:03,857 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [124fad3d] Correlation ID: 124fad3d, Job ID: 67503623-8719-4e8d-beca-1f91d623a6d9, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:13:03,864 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [124fad3d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:13:03,866 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-71) [124fad3d] Correlation ID: 5b63ef7, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:13:03,872 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-71) [26da4f7a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:13:03,873 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-71) [29ea26d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:13:03,873 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-71) [29ea26d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) [49239c0f] Autorecovering 1 hosts >2014-04-30 08:15:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) [49239c0f] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:15:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:15:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7d5d89ed >2014-04-30 08:15:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] FINISH, SetVdsStatusVDSCommand, log id: 7d5d89ed >2014-04-30 08:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 274bca93 >2014-04-30 08:15:00,470 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 305c7890 >2014-04-30 08:15:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] Failed in GetHardwareInfoVDS method >2014-04-30 08:15:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:15:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6ef4ebe >2014-04-30 08:15:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] HostName = 10.34.27.209 >2014-04-30 08:15:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:15:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [211b7dd1] FINISH, GetHardwareInfoVDSCommand, log id: 305c7890 >2014-04-30 08:15:00,699 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [211b7dd1] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:15:00,710 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-90) [4cb18351] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:15:00,711 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-90) [3cb94118] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:15:00,712 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) [3cb94118] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 274bca93 >2014-04-30 08:15:00,728 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [3cb94118] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:15:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [3cb94118] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:15:01,262 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 22f67bf3 >2014-04-30 08:15:01,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] Failed in GetHardwareInfoVDS method >2014-04-30 08:15:01,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:15:01,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@6443a073 >2014-04-30 08:15:01,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] HostName = 10.34.27.209 >2014-04-30 08:15:01,481 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:15:01,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-61) [408b3dcd] FINISH, GetHardwareInfoVDSCommand, log id: 22f67bf3 >2014-04-30 08:15:01,487 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [408b3dcd] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:15:01,496 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [408b3dcd] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:15:01,715 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-61) [5b0468fc] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:15:01,717 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-61) [3728854a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:15:01,723 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-61) [3728854a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 363b98f2 >2014-04-30 08:18:01,725 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-61) [3728854a] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:18:01,725 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-61) [3728854a] FINISH, ConnectStorageServerVDSCommand, log id: 363b98f2 >2014-04-30 08:18:01,725 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-61) [3728854a] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:18:01,728 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 29269df5 >2014-04-30 08:18:02,180 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 29269df5 >2014-04-30 08:18:02,180 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:18:02,191 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-61) [52d61a49] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:18:02,192 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) [52d61a49] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4f4ed09f >2014-04-30 08:18:02,194 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-61) [52d61a49] FINISH, SetVdsStatusVDSCommand, log id: 4f4ed09f >2014-04-30 08:18:02,201 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [52d61a49] Correlation ID: 52d61a49, Job ID: 0841ecc2-dd5d-4c4b-8cb4-3ea21a04135c, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:18:02,208 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [52d61a49] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:18:02,210 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-61) [52d61a49] Correlation ID: 5b0468fc, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:18:02,214 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-61) [66ed470] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:18:02,215 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-61) [56476bf7] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:18:02,216 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-61) [56476bf7] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [483bbf17] Autorecovering 1 hosts >2014-04-30 08:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-58) [483bbf17] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:20:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4e75839a >2014-04-30 08:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] FINISH, SetVdsStatusVDSCommand, log id: 4e75839a >2014-04-30 08:20:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1d64c6df >2014-04-30 08:20:00,469 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4adb9ea8 >2014-04-30 08:20:00,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] Failed in GetHardwareInfoVDS method >2014-04-30 08:20:00,689 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:20:00,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@188d4dfd >2014-04-30 08:20:00,689 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] HostName = 10.34.27.209 >2014-04-30 08:20:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:20:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-58) [5534c01c] FINISH, GetHardwareInfoVDSCommand, log id: 4adb9ea8 >2014-04-30 08:20:00,697 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-58) [5534c01c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:20:00,707 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-58) [35aaf89b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:20:00,708 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-58) [4de86ef1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:20:00,709 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-58) [4de86ef1] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1d64c6df >2014-04-30 08:20:00,722 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [4de86ef1] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:20:00,723 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-58) [4de86ef1] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:20:02,572 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [61de2f6e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2eecf165 >2014-04-30 08:20:02,794 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [61de2f6e] Failed in GetHardwareInfoVDS method >2014-04-30 08:20:02,794 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [61de2f6e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:20:02,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [61de2f6e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@45dc35f4 >2014-04-30 08:20:02,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [61de2f6e] HostName = 10.34.27.209 >2014-04-30 08:20:02,794 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [61de2f6e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:20:02,794 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-27) [61de2f6e] FINISH, GetHardwareInfoVDSCommand, log id: 2eecf165 >2014-04-30 08:20:02,799 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [61de2f6e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:20:02,808 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [61de2f6e] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:20:03,027 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-27) [4c85e46d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:20:03,029 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-27) [5d67cb4d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:20:03,035 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-27) [5d67cb4d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1707894b >2014-04-30 08:23:03,037 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-27) [5d67cb4d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:23:03,037 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-27) [5d67cb4d] FINISH, ConnectStorageServerVDSCommand, log id: 1707894b >2014-04-30 08:23:03,037 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-27) [5d67cb4d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:23:03,042 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 20e6a6db >2014-04-30 08:23:03,486 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 20e6a6db >2014-04-30 08:23:03,486 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:23:03,498 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-27) [5b97b0d1] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:23:03,500 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) [5b97b0d1] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3945df70 >2014-04-30 08:23:03,503 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-27) [5b97b0d1] FINISH, SetVdsStatusVDSCommand, log id: 3945df70 >2014-04-30 08:23:03,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [5b97b0d1] Correlation ID: 5b97b0d1, Job ID: 31f43d9e-8300-4a17-aeb8-7545fced9a55, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:23:03,517 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [5b97b0d1] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:23:03,519 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-27) [5b97b0d1] Correlation ID: 4c85e46d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:23:03,523 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-27) [7c315259] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:23:03,524 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-27) [4bd6b176] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:23:03,525 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-27) [4bd6b176] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-39) [4d62b41d] Autorecovering 1 hosts >2014-04-30 08:25:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-39) [4d62b41d] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:25:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:25:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 2587f888 >2014-04-30 08:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] FINISH, SetVdsStatusVDSCommand, log id: 2587f888 >2014-04-30 08:25:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7bfc616f >2014-04-30 08:25:00,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6bcbc0cf >2014-04-30 08:25:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] Failed in GetHardwareInfoVDS method >2014-04-30 08:25:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:25:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@ddf8d53 >2014-04-30 08:25:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] HostName = 10.34.27.209 >2014-04-30 08:25:00,694 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:25:00,694 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-39) [705cb3dd] FINISH, GetHardwareInfoVDSCommand, log id: 6bcbc0cf >2014-04-30 08:25:00,714 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-39) [705cb3dd] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:25:00,729 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-39) [8ca51fe] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:25:00,731 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-39) [75a777c0] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:25:00,732 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-39) [75a777c0] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7bfc616f >2014-04-30 08:25:00,747 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-39) [75a777c0] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:25:00,749 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-39) [75a777c0] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:25:03,881 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 587272e6 >2014-04-30 08:25:04,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] Failed in GetHardwareInfoVDS method >2014-04-30 08:25:04,100 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:25:04,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@41d58767 >2014-04-30 08:25:04,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] HostName = 10.34.27.209 >2014-04-30 08:25:04,101 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:25:04,101 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] FINISH, GetHardwareInfoVDSCommand, log id: 587272e6 >2014-04-30 08:25:04,105 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:25:04,115 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [45ddd7f3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:25:04,336 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-68) [446c604d] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:25:04,337 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [b950413] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:25:04,343 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [b950413] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4be1e6d1 >2014-04-30 08:28:04,345 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [b950413] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:28:04,345 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-68) [b950413] FINISH, ConnectStorageServerVDSCommand, log id: 4be1e6d1 >2014-04-30 08:28:04,345 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-68) [b950413] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:28:04,349 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 282ccda6 >2014-04-30 08:28:04,792 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 282ccda6 >2014-04-30 08:28:04,792 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:28:04,803 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-68) [45c8a597] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:28:04,804 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [45c8a597] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4e7f0f3e >2014-04-30 08:28:04,806 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-68) [45c8a597] FINISH, SetVdsStatusVDSCommand, log id: 4e7f0f3e >2014-04-30 08:28:04,812 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [45c8a597] Correlation ID: 45c8a597, Job ID: b863744f-be9b-4b0b-af17-014ade4526ed, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:28:04,820 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [45c8a597] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:28:04,822 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-68) [45c8a597] Correlation ID: 446c604d, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:28:04,828 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-68) [665c2f37] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:28:04,829 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-68) [242be080] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:28:04,830 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-68) [242be080] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-6) [3f865792] Autorecovering 1 hosts >2014-04-30 08:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-6) [3f865792] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:30:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:30:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 21e4e8e >2014-04-30 08:30:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] FINISH, SetVdsStatusVDSCommand, log id: 21e4e8e >2014-04-30 08:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 734abca4 >2014-04-30 08:30:00,468 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 542f60a2 >2014-04-30 08:30:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] Failed in GetHardwareInfoVDS method >2014-04-30 08:30:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:30:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@399f73bf >2014-04-30 08:30:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] HostName = 10.34.27.209 >2014-04-30 08:30:00,688 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:30:00,688 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-6) [58393dfa] FINISH, GetHardwareInfoVDSCommand, log id: 542f60a2 >2014-04-30 08:30:00,694 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-6) [58393dfa] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:30:00,702 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-6) [c7b41d2] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:30:00,704 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-6) [768225c1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:30:00,704 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-6) [768225c1] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 734abca4 >2014-04-30 08:30:00,719 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [768225c1] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:30:00,720 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-6) [768225c1] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:30:02,190 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 20b67c00 >2014-04-30 08:30:02,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] Failed in GetHardwareInfoVDS method >2014-04-30 08:30:02,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:30:02,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2c86f4c7 >2014-04-30 08:30:02,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] HostName = 10.34.27.209 >2014-04-30 08:30:02,411 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:30:02,411 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-4) [7ff065d4] FINISH, GetHardwareInfoVDSCommand, log id: 20b67c00 >2014-04-30 08:30:02,416 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [7ff065d4] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:30:02,426 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [7ff065d4] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:30:02,647 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-4) [772431a6] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:30:02,649 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-4) [7becd6c2] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:30:02,656 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [7becd6c2] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 2fb49ba0 >2014-04-30 08:33:02,657 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [7becd6c2] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:33:02,657 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-4) [7becd6c2] FINISH, ConnectStorageServerVDSCommand, log id: 2fb49ba0 >2014-04-30 08:33:02,658 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-4) [7becd6c2] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:33:02,661 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: c57b323 >2014-04-30 08:33:03,100 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: c57b323 >2014-04-30 08:33:03,100 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:33:03,111 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-4) [61a03520] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:33:03,113 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [61a03520] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 21b7e99f >2014-04-30 08:33:03,125 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-4) [61a03520] FINISH, SetVdsStatusVDSCommand, log id: 21b7e99f >2014-04-30 08:33:03,141 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [61a03520] Correlation ID: 61a03520, Job ID: fed8b78c-6669-4f41-950e-3528bf40ffba, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:33:03,165 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [61a03520] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:33:03,167 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-4) [61a03520] Correlation ID: 772431a6, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:33:03,172 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-4) [d3c947b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:33:03,173 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-4) [61ef4c61] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:33:03,174 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-4) [61ef4c61] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) [2e159e5e] Autorecovering 1 hosts >2014-04-30 08:35:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-54) [2e159e5e] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:35:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:35:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 1455191f >2014-04-30 08:35:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] FINISH, SetVdsStatusVDSCommand, log id: 1455191f >2014-04-30 08:35:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5da3565 >2014-04-30 08:35:00,484 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 77303ce3 >2014-04-30 08:35:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] Failed in GetHardwareInfoVDS method >2014-04-30 08:35:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:35:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4a55e78e >2014-04-30 08:35:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] HostName = 10.34.27.209 >2014-04-30 08:35:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:35:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] FINISH, GetHardwareInfoVDSCommand, log id: 77303ce3 >2014-04-30 08:35:00,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-54) [10c7d1d0] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:35:00,720 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-54) [14e1bde4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:35:00,721 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-54) [75b7aaa5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:35:00,722 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-54) [75b7aaa5] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5da3565 >2014-04-30 08:35:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [75b7aaa5] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:35:00,738 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-54) [75b7aaa5] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:35:03,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [42b0efd0] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 146c72f0 >2014-04-30 08:35:03,780 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [42b0efd0] Failed in GetHardwareInfoVDS method >2014-04-30 08:35:03,780 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [42b0efd0] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:35:03,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [42b0efd0] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5bedf509 >2014-04-30 08:35:03,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [42b0efd0] HostName = 10.34.27.209 >2014-04-30 08:35:03,780 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [42b0efd0] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:35:03,780 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-45) [42b0efd0] FINISH, GetHardwareInfoVDSCommand, log id: 146c72f0 >2014-04-30 08:35:03,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [42b0efd0] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:35:03,794 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [42b0efd0] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:35:04,013 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-45) [ac10041] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:35:04,015 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [28c2c865] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:35:04,021 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [28c2c865] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 47852b9e >2014-04-30 08:36:05,028 INFO [org.ovirt.engine.core.bll.DbUserCacheManager] (DefaultQuartzScheduler_Worker-66) [4400bde0] Start refreshing all users data >2014-04-30 08:36:07,319 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) [10535fa3] Attempting to update VMs/Templates Ovf. >2014-04-30 08:36:07,320 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) [10535fa3] Attempting to update VM OVFs in Data Center NFS >2014-04-30 08:36:07,321 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) [10535fa3] Successfully updated VM OVFs in Data Center NFS >2014-04-30 08:36:07,322 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) [10535fa3] Attempting to update template OVFs in Data Center NFS >2014-04-30 08:36:07,323 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) [10535fa3] Successfully updated templates OVFs in Data Center NFS >2014-04-30 08:36:07,323 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) [10535fa3] Attempting to remove unneeded template/vm OVFs in Data Center NFS >2014-04-30 08:36:07,323 INFO [org.ovirt.engine.core.bll.OvfDataUpdater] (DefaultQuartzScheduler_Worker-80) [10535fa3] Successfully removed unneeded template/vm OVFs in Data Center NFS >2014-04-30 08:38:04,023 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [28c2c865] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:38:04,023 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-45) [28c2c865] FINISH, ConnectStorageServerVDSCommand, log id: 47852b9e >2014-04-30 08:38:04,023 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-45) [28c2c865] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:38:04,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: bd2938f >2014-04-30 08:38:04,464 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-46) FINISH, ConnectStoragePoolVDSCommand, log id: bd2938f >2014-04-30 08:38:04,464 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-46) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:38:04,476 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-45) [efe0d37] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:38:04,477 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [efe0d37] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 72bc8d4e >2014-04-30 08:38:04,480 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-45) [efe0d37] FINISH, SetVdsStatusVDSCommand, log id: 72bc8d4e >2014-04-30 08:38:04,486 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [efe0d37] Correlation ID: efe0d37, Job ID: 408b428e-8f63-4aad-80e7-edbe87774c4e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:38:04,492 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [efe0d37] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:38:04,495 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-45) [efe0d37] Correlation ID: ac10041, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:38:04,500 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-45) [5fa80c59] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:38:04,501 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-45) [22f2a4eb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:38:04,501 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-45) [22f2a4eb] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) [3cb94118] Autorecovering 1 hosts >2014-04-30 08:40:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-90) [3cb94118] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:40:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:40:00,007 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 776d6a4c >2014-04-30 08:40:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] FINISH, SetVdsStatusVDSCommand, log id: 776d6a4c >2014-04-30 08:40:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 7de2e9e2 >2014-04-30 08:40:00,467 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 54c24cc >2014-04-30 08:40:00,685 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] Failed in GetHardwareInfoVDS method >2014-04-30 08:40:00,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:40:00,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4a03cfcb >2014-04-30 08:40:00,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] HostName = 10.34.27.209 >2014-04-30 08:40:00,686 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:40:00,686 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-90) [6b85123e] FINISH, GetHardwareInfoVDSCommand, log id: 54c24cc >2014-04-30 08:40:00,691 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-90) [6b85123e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:40:00,700 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-90) [20cc0440] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:40:00,702 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-90) [1f68809] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:40:00,702 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-90) [1f68809] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 7de2e9e2 >2014-04-30 08:40:00,715 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [1f68809] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:40:00,716 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-90) [1f68809] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:40:01,862 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 3a8b32b6 >2014-04-30 08:40:02,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] Failed in GetHardwareInfoVDS method >2014-04-30 08:40:02,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:40:02,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@8ae68c1 >2014-04-30 08:40:02,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] HostName = 10.34.27.209 >2014-04-30 08:40:02,081 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:40:02,081 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] FINISH, GetHardwareInfoVDSCommand, log id: 3a8b32b6 >2014-04-30 08:40:02,086 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:40:02,096 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [5e1db2c8] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:40:02,314 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-55) [3f100c6f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:40:02,316 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [77928d3d] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:40:02,322 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [77928d3d] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 720e672 >2014-04-30 08:43:02,323 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [77928d3d] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:43:02,323 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-55) [77928d3d] FINISH, ConnectStorageServerVDSCommand, log id: 720e672 >2014-04-30 08:43:02,324 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-55) [77928d3d] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:43:02,327 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 22ac19cf >2014-04-30 08:43:02,761 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 22ac19cf >2014-04-30 08:43:02,761 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:43:02,772 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-55) [76b3f64f] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:43:02,773 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [76b3f64f] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3c5a9ccb >2014-04-30 08:43:02,776 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-55) [76b3f64f] FINISH, SetVdsStatusVDSCommand, log id: 3c5a9ccb >2014-04-30 08:43:02,783 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [76b3f64f] Correlation ID: 76b3f64f, Job ID: dcea7b68-9b65-4e13-b632-d7bf7ba3f00b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:43:02,791 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [76b3f64f] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:43:02,794 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-55) [76b3f64f] Correlation ID: 3f100c6f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:43:02,798 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-55) [3d68d157] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:43:02,799 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-55) [51afc72f] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:43:02,800 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-55) [51afc72f] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:45:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-95) [179ebe56] Autorecovering 1 hosts >2014-04-30 08:45:00,006 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-95) [179ebe56] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:45:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:45:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 65392fa1 >2014-04-30 08:45:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] FINISH, SetVdsStatusVDSCommand, log id: 65392fa1 >2014-04-30 08:45:00,016 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: fb2b661 >2014-04-30 08:45:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 31c110d7 >2014-04-30 08:45:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] Failed in GetHardwareInfoVDS method >2014-04-30 08:45:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:45:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7a38acc5 >2014-04-30 08:45:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] HostName = 10.34.27.209 >2014-04-30 08:45:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:45:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-95) [751da8b2] FINISH, GetHardwareInfoVDSCommand, log id: 31c110d7 >2014-04-30 08:45:00,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-95) [751da8b2] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:45:00,713 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-95) [601a76d7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:45:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-95) [6a007fad] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:45:00,716 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-95) [6a007fad] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: fb2b661 >2014-04-30 08:45:00,732 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [6a007fad] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:45:00,733 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-95) [6a007fad] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:45:03,165 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [5b883a5a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6535766 >2014-04-30 08:45:03,385 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [5b883a5a] Failed in GetHardwareInfoVDS method >2014-04-30 08:45:03,385 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [5b883a5a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:45:03,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [5b883a5a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@498eecc8 >2014-04-30 08:45:03,385 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [5b883a5a] HostName = 10.34.27.209 >2014-04-30 08:45:03,386 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [5b883a5a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:45:03,386 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [5b883a5a] FINISH, GetHardwareInfoVDSCommand, log id: 6535766 >2014-04-30 08:45:03,391 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [5b883a5a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:45:03,403 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [5b883a5a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:45:03,623 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-59) [2208bcd] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:45:03,625 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-59) [621c492a] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:45:03,631 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [621c492a] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 3442ad31 >2014-04-30 08:48:03,632 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [621c492a] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:48:03,632 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-59) [621c492a] FINISH, ConnectStorageServerVDSCommand, log id: 3442ad31 >2014-04-30 08:48:03,632 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-59) [621c492a] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:48:03,636 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7a8f4be8 >2014-04-30 08:48:04,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 7a8f4be8 >2014-04-30 08:48:04,070 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:48:04,081 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-59) [464f9cbc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:48:04,082 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [464f9cbc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 531a9ad2 >2014-04-30 08:48:04,085 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [464f9cbc] FINISH, SetVdsStatusVDSCommand, log id: 531a9ad2 >2014-04-30 08:48:04,091 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [464f9cbc] Correlation ID: 464f9cbc, Job ID: 72960f06-b6a2-4a61-b422-572aee2d7ce3, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:48:04,099 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [464f9cbc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:48:04,102 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [464f9cbc] Correlation ID: 2208bcd, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:48:04,106 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-59) [280f36e7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:48:04,107 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-59) [2c46419] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:48:04,107 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-59) [2c46419] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) [7b54e090] Autorecovering 1 hosts >2014-04-30 08:50:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-8) [7b54e090] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:50:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:50:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 45eafeb7 >2014-04-30 08:50:00,011 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] FINISH, SetVdsStatusVDSCommand, log id: 45eafeb7 >2014-04-30 08:50:00,012 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 333f8949 >2014-04-30 08:50:00,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2a39ea >2014-04-30 08:50:00,691 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] Failed in GetHardwareInfoVDS method >2014-04-30 08:50:00,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:50:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5506018a >2014-04-30 08:50:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] HostName = 10.34.27.209 >2014-04-30 08:50:00,692 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:50:00,692 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] FINISH, GetHardwareInfoVDSCommand, log id: 2a39ea >2014-04-30 08:50:00,697 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-8) [2fdc2d15] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:50:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-8) [5ffd899] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:50:00,710 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-8) [4b0e1f99] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:50:00,711 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-8) [4b0e1f99] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 333f8949 >2014-04-30 08:50:00,730 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [4b0e1f99] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:50:00,731 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-8) [4b0e1f99] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:50:01,460 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [482631c] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 52312258 >2014-04-30 08:50:01,679 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [482631c] Failed in GetHardwareInfoVDS method >2014-04-30 08:50:01,679 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [482631c] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:50:01,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [482631c] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@17c5e990 >2014-04-30 08:50:01,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [482631c] HostName = 10.34.27.209 >2014-04-30 08:50:01,679 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [482631c] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:50:01,679 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-62) [482631c] FINISH, GetHardwareInfoVDSCommand, log id: 52312258 >2014-04-30 08:50:01,684 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [482631c] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:50:01,693 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [482631c] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:50:01,913 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-62) [c166493] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:50:01,915 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-62) [81ef91c] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:50:01,921 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [81ef91c] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: ebc90f9 >2014-04-30 08:53:01,922 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [81ef91c] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:53:01,922 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-62) [81ef91c] FINISH, ConnectStorageServerVDSCommand, log id: ebc90f9 >2014-04-30 08:53:01,923 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-62) [81ef91c] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:53:01,927 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 65d7f15c >2014-04-30 08:53:02,376 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 65d7f15c >2014-04-30 08:53:02,376 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:53:02,390 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-62) [43da4b29] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:53:02,391 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [43da4b29] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 1d9cbcf3 >2014-04-30 08:53:02,393 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-62) [43da4b29] FINISH, SetVdsStatusVDSCommand, log id: 1d9cbcf3 >2014-04-30 08:53:02,398 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [43da4b29] Correlation ID: 43da4b29, Job ID: fbd76f43-c55e-4432-803f-e397bf1b740e, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:53:02,405 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [43da4b29] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:53:02,407 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-62) [43da4b29] Correlation ID: c166493, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:53:02,411 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-62) [7afebde0] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:53:02,412 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-62) [567b0cb2] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:53:02,413 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-62) [567b0cb2] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 08:55:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-93) [629a4741] Autorecovering 1 hosts >2014-04-30 08:55:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-93) [629a4741] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 08:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 08:55:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 08:55:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3b52d787 >2014-04-30 08:55:00,015 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] FINISH, SetVdsStatusVDSCommand, log id: 3b52d787 >2014-04-30 08:55:00,017 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 2e37f81d >2014-04-30 08:55:00,475 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6ecd324a >2014-04-30 08:55:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] Failed in GetHardwareInfoVDS method >2014-04-30 08:55:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:55:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@61a0d11a >2014-04-30 08:55:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] HostName = 10.34.27.209 >2014-04-30 08:55:00,695 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:55:00,695 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-93) [73ec1dca] FINISH, GetHardwareInfoVDSCommand, log id: 6ecd324a >2014-04-30 08:55:00,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-93) [73ec1dca] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:55:00,714 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-93) [44f36e18] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:55:00,715 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-93) [1fb1049a] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:55:00,715 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-93) [1fb1049a] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 2e37f81d >2014-04-30 08:55:00,739 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [1fb1049a] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 08:55:00,741 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-93) [1fb1049a] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 08:55:02,762 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5a9a4f37 >2014-04-30 08:55:02,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] Failed in GetHardwareInfoVDS method >2014-04-30 08:55:02,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:55:02,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@48c6d259 >2014-04-30 08:55:02,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] HostName = 10.34.27.209 >2014-04-30 08:55:02,984 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 08:55:02,984 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-85) [2ad53db9] FINISH, GetHardwareInfoVDSCommand, log id: 5a9a4f37 >2014-04-30 08:55:02,989 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [2ad53db9] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 08:55:02,998 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [2ad53db9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 08:55:03,217 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-85) [15052292] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:55:03,218 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-85) [2f594e77] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 08:55:03,224 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-85) [2f594e77] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 649428c1 >2014-04-30 08:58:03,225 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-85) [2f594e77] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 08:58:03,225 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-85) [2f594e77] FINISH, ConnectStorageServerVDSCommand, log id: 649428c1 >2014-04-30 08:58:03,226 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-85) [2f594e77] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 08:58:03,230 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 35bae4c9 >2014-04-30 08:58:03,674 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 35bae4c9 >2014-04-30 08:58:03,674 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 08:58:03,686 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-85) [15b0371a] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:58:03,687 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [15b0371a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 3675b4a2 >2014-04-30 08:58:03,689 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-85) [15b0371a] FINISH, SetVdsStatusVDSCommand, log id: 3675b4a2 >2014-04-30 08:58:03,694 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [15b0371a] Correlation ID: 15b0371a, Job ID: ead810f7-0d9b-4844-9a7b-5e780cd94bbb, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 08:58:03,702 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [15b0371a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 08:58:03,704 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-85) [15b0371a] Correlation ID: 15052292, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 08:58:03,710 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-85) [38a80063] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:58:03,711 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-85) [798ff75d] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 08:58:03,711 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-85) [798ff75d] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 09:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-43) [2ddda5a1] Autorecovering 1 hosts >2014-04-30 09:00:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-43) [2ddda5a1] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 09:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 09:00:00,007 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 09:00:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 6e4ae357 >2014-04-30 09:00:00,013 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] FINISH, SetVdsStatusVDSCommand, log id: 6e4ae357 >2014-04-30 09:00:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 53e6b02e >2014-04-30 09:00:00,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5bb50a6d >2014-04-30 09:00:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] Failed in GetHardwareInfoVDS method >2014-04-30 09:00:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:00:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@4abfa924 >2014-04-30 09:00:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] HostName = 10.34.27.209 >2014-04-30 09:00:00,693 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:00:00,693 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-43) [426ac5e7] FINISH, GetHardwareInfoVDSCommand, log id: 5bb50a6d >2014-04-30 09:00:00,700 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-43) [426ac5e7] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:00:00,720 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-43) [63815e12] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:00:00,721 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-43) [27ffa3f8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:00:00,722 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-43) [27ffa3f8] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 53e6b02e >2014-04-30 09:00:00,736 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [27ffa3f8] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 09:00:00,737 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-43) [27ffa3f8] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:00:01,070 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 37b023bf >2014-04-30 09:00:01,291 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] Failed in GetHardwareInfoVDS method >2014-04-30 09:00:01,291 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:00:01,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@59e982ec >2014-04-30 09:00:01,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] HostName = 10.34.27.209 >2014-04-30 09:00:01,291 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:00:01,291 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-78) [6ca36e68] FINISH, GetHardwareInfoVDSCommand, log id: 37b023bf >2014-04-30 09:00:01,295 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [6ca36e68] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:00:01,304 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [6ca36e68] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 09:00:01,523 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-78) [3811aa28] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:00:01,525 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [60b504fc] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:00:01,531 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [60b504fc] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1a5aa59d >2014-04-30 09:03:01,532 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [60b504fc] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 09:03:01,532 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-78) [60b504fc] FINISH, ConnectStorageServerVDSCommand, log id: 1a5aa59d >2014-04-30 09:03:01,532 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-78) [60b504fc] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 09:03:01,536 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 42efe8c7 >2014-04-30 09:03:02,024 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-41) FINISH, ConnectStoragePoolVDSCommand, log id: 42efe8c7 >2014-04-30 09:03:02,024 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-41) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 09:03:02,034 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-78) [3abe7bc] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:03:02,035 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [3abe7bc] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7daca258 >2014-04-30 09:03:02,038 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-78) [3abe7bc] FINISH, SetVdsStatusVDSCommand, log id: 7daca258 >2014-04-30 09:03:02,054 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [3abe7bc] Correlation ID: 3abe7bc, Job ID: eb307fd6-d26d-4dd8-a3f4-a2c9fdc712be, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 09:03:02,070 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [3abe7bc] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 09:03:02,075 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-78) [3abe7bc] Correlation ID: 3811aa28, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 09:03:02,087 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-78) [55d9ba46] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:03:02,088 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-78) [151edee] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:03:02,089 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-78) [151edee] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 09:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-49) [5a770a75] Autorecovering 1 hosts >2014-04-30 09:05:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-49) [5a770a75] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 09:05:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 09:05:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 09:05:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 3ec6485d >2014-04-30 09:05:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] FINISH, SetVdsStatusVDSCommand, log id: 3ec6485d >2014-04-30 09:05:00,013 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 4c05ff68 >2014-04-30 09:05:00,528 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 461633f8 >2014-04-30 09:05:00,749 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] Failed in GetHardwareInfoVDS method >2014-04-30 09:05:00,749 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:05:00,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@7316529d >2014-04-30 09:05:00,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] HostName = 10.34.27.209 >2014-04-30 09:05:00,749 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:05:00,749 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-49) [5443982e] FINISH, GetHardwareInfoVDSCommand, log id: 461633f8 >2014-04-30 09:05:00,754 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-49) [5443982e] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:05:00,763 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-49) [317ae0e] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:05:00,764 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-49) [5b5c50f1] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:05:00,764 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-49) [5b5c50f1] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 4c05ff68 >2014-04-30 09:05:00,779 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5b5c50f1] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 09:05:00,780 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-49) [5b5c50f1] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:05:02,479 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4120f15 >2014-04-30 09:05:02,713 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] Failed in GetHardwareInfoVDS method >2014-04-30 09:05:02,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:05:02,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@231f1c27 >2014-04-30 09:05:02,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] HostName = 10.34.27.209 >2014-04-30 09:05:02,714 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:05:02,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-64) [5cc06a25] FINISH, GetHardwareInfoVDSCommand, log id: 4120f15 >2014-04-30 09:05:02,718 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [5cc06a25] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:05:02,728 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [5cc06a25] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 09:05:02,969 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-64) [7ac5f25c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:05:02,971 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [4d0621ba] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:05:02,977 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [4d0621ba] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 1c8572bf >2014-04-30 09:08:02,978 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [4d0621ba] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 09:08:02,978 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-64) [4d0621ba] FINISH, ConnectStorageServerVDSCommand, log id: 1c8572bf >2014-04-30 09:08:02,979 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-64) [4d0621ba] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 09:08:02,988 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 60e91284 >2014-04-30 09:08:03,424 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-49) FINISH, ConnectStoragePoolVDSCommand, log id: 60e91284 >2014-04-30 09:08:03,424 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-49) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 09:08:03,435 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-64) [2e74028d] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:08:03,435 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [2e74028d] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 18820409 >2014-04-30 09:08:03,438 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-64) [2e74028d] FINISH, SetVdsStatusVDSCommand, log id: 18820409 >2014-04-30 09:08:03,444 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [2e74028d] Correlation ID: 2e74028d, Job ID: 5087bb7c-7d20-46ec-b0bc-0e30fd5f24ac, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 09:08:03,450 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [2e74028d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 09:08:03,452 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-64) [2e74028d] Correlation ID: 7ac5f25c, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 09:08:03,456 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-64) [1244727b] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:08:03,457 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-64) [bf06694] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:08:03,458 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-64) [bf06694] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 09:10:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-60) [4b138017] Autorecovering 1 hosts >2014-04-30 09:10:00,005 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-60) [4b138017] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 09:10:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [14921077] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:10:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [14921077] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:10:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [14921077] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 09:10:00,010 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [14921077] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 09:10:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [14921077] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 753c916f >2014-04-30 09:10:00,025 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-60) [14921077] FINISH, SetVdsStatusVDSCommand, log id: 753c916f >2014-04-30 09:10:00,028 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) [14921077] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 6bdb52c7 >2014-04-30 09:10:00,495 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [14921077] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 55a2e610 >2014-04-30 09:10:00,716 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [14921077] Failed in GetHardwareInfoVDS method >2014-04-30 09:10:00,716 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [14921077] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:10:00,716 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [14921077] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@39ae505c >2014-04-30 09:10:00,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [14921077] HostName = 10.34.27.209 >2014-04-30 09:10:00,717 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [14921077] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:10:00,717 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-60) [14921077] FINISH, GetHardwareInfoVDSCommand, log id: 55a2e610 >2014-04-30 09:10:00,730 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-60) [14921077] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:10:00,743 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-60) [4854a3d6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:10:00,744 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-60) [490b91e8] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:10:00,745 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-60) [490b91e8] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 6bdb52c7 >2014-04-30 09:10:00,762 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [490b91e8] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 09:10:00,763 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-60) [490b91e8] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:10:03,867 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 2fc0e3e1 >2014-04-30 09:10:04,154 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] Failed in GetHardwareInfoVDS method >2014-04-30 09:10:04,155 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:10:04,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@20b0e145 >2014-04-30 09:10:04,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] HostName = 10.34.27.209 >2014-04-30 09:10:04,155 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:10:04,155 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] FINISH, GetHardwareInfoVDSCommand, log id: 2fc0e3e1 >2014-04-30 09:10:04,160 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:10:04,177 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [2cb2df4d] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 09:10:04,404 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-73) [522fef4c] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:10:04,406 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [6787e0ee] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:10:04,412 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [6787e0ee] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 5e9f5c46 >2014-04-30 09:13:04,413 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [6787e0ee] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 09:13:04,413 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-73) [6787e0ee] FINISH, ConnectStorageServerVDSCommand, log id: 5e9f5c46 >2014-04-30 09:13:04,413 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-73) [6787e0ee] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 09:13:04,417 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 791798c >2014-04-30 09:13:04,857 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 791798c >2014-04-30 09:13:04,857 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 09:13:04,868 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-73) [3b052079] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:13:04,869 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [3b052079] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 11667f27 >2014-04-30 09:13:04,871 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-73) [3b052079] FINISH, SetVdsStatusVDSCommand, log id: 11667f27 >2014-04-30 09:13:04,878 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [3b052079] Correlation ID: 3b052079, Job ID: 3987d81f-b157-4498-93d2-8075bc592c9a, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 09:13:04,886 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [3b052079] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 09:13:04,896 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-73) [3b052079] Correlation ID: 522fef4c, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 09:13:04,901 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-73) [14fcd7f7] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:13:04,902 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-73) [7cde43e9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:13:04,903 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-73) [7cde43e9] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 09:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-31) [38bf1fe4] Autorecovering 1 hosts >2014-04-30 09:15:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-31) [38bf1fe4] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 09:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 09:15:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 09:15:00,009 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4855f7a8 >2014-04-30 09:15:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] FINISH, SetVdsStatusVDSCommand, log id: 4855f7a8 >2014-04-30 09:15:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 1cf6ba5a >2014-04-30 09:15:00,471 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 36f322a5 >2014-04-30 09:15:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] Failed in GetHardwareInfoVDS method >2014-04-30 09:15:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:15:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@14679ad2 >2014-04-30 09:15:00,690 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] HostName = 10.34.27.209 >2014-04-30 09:15:00,690 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:15:00,691 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-31) [520bb3a3] FINISH, GetHardwareInfoVDSCommand, log id: 36f322a5 >2014-04-30 09:15:00,696 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-31) [520bb3a3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:15:00,708 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-31) [5bc2479f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:15:00,709 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-31) [7cbc05fb] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:15:00,710 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-31) [7cbc05fb] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 1cf6ba5a >2014-04-30 09:15:00,726 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [7cbc05fb] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 09:15:00,727 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-31) [7cbc05fb] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:15:02,254 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 5aab787b >2014-04-30 09:15:02,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] Failed in GetHardwareInfoVDS method >2014-04-30 09:15:02,473 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:15:02,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@2f7301be >2014-04-30 09:15:02,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] HostName = 10.34.27.209 >2014-04-30 09:15:02,474 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:15:02,474 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-98) [111f2de] FINISH, GetHardwareInfoVDSCommand, log id: 5aab787b >2014-04-30 09:15:02,478 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [111f2de] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:15:02,487 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [111f2de] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 09:15:02,705 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-98) [28b6a41f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:15:02,707 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [19a9006b] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:15:02,714 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [19a9006b] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 755ecbf7 >2014-04-30 09:18:02,715 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [19a9006b] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 09:18:02,715 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-98) [19a9006b] FINISH, ConnectStorageServerVDSCommand, log id: 755ecbf7 >2014-04-30 09:18:02,715 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-98) [19a9006b] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 09:18:02,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 33388fe5 >2014-04-30 09:18:03,171 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 33388fe5 >2014-04-30 09:18:03,171 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 09:18:03,183 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-98) [1bd713d9] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:18:03,184 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [1bd713d9] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 7ae6345e >2014-04-30 09:18:03,187 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-98) [1bd713d9] FINISH, SetVdsStatusVDSCommand, log id: 7ae6345e >2014-04-30 09:18:03,193 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [1bd713d9] Correlation ID: 1bd713d9, Job ID: 1d1bb3cf-2899-45de-bd22-e07aeb1d3d6b, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 09:18:03,200 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [1bd713d9] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 09:18:03,202 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-98) [1bd713d9] Correlation ID: 28b6a41f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 09:18:03,206 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-98) [60e26728] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:18:03,207 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-98) [2db26ca9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:18:03,207 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-98) [2db26ca9] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 09:20:00,003 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [49ae3be3] Autorecovering 1 hosts >2014-04-30 09:20:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-20) [49ae3be3] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 09:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [361df663] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [361df663] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [361df663] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 09:20:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [361df663] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 09:20:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [361df663] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 45300cc8 >2014-04-30 09:20:00,010 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-20) [361df663] FINISH, SetVdsStatusVDSCommand, log id: 45300cc8 >2014-04-30 09:20:00,011 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [361df663] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 78724bed >2014-04-30 09:20:05,594 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [361df663] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 33b670ad >2014-04-30 09:20:05,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [361df663] Failed in GetHardwareInfoVDS method >2014-04-30 09:20:05,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [361df663] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:20:05,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [361df663] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@e9f8861 >2014-04-30 09:20:05,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [361df663] HostName = 10.34.27.209 >2014-04-30 09:20:05,814 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [361df663] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:20:05,814 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-20) [361df663] FINISH, GetHardwareInfoVDSCommand, log id: 33b670ad >2014-04-30 09:20:05,819 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-20) [361df663] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:20:05,829 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-20) [55dd737a] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:20:05,830 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-20) [686275a5] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:20:05,831 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-20) [686275a5] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 78724bed >2014-04-30 09:20:05,847 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [686275a5] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 09:20:05,848 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-20) [686275a5] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:20:06,560 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [1c82b147] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4ea854c9 >2014-04-30 09:20:06,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [1c82b147] Failed in GetHardwareInfoVDS method >2014-04-30 09:20:06,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [1c82b147] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:20:06,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [1c82b147] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@5e629b76 >2014-04-30 09:20:06,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [1c82b147] HostName = 10.34.27.209 >2014-04-30 09:20:06,782 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [1c82b147] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:20:06,782 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-91) [1c82b147] FINISH, GetHardwareInfoVDSCommand, log id: 4ea854c9 >2014-04-30 09:20:06,786 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [1c82b147] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:20:06,796 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [1c82b147] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 09:20:07,016 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-91) [6fad2fa1] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:20:07,017 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [3f17578] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:20:07,027 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [3f17578] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 205bdce >2014-04-30 09:23:07,028 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [3f17578] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 09:23:07,028 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-91) [3f17578] FINISH, ConnectStorageServerVDSCommand, log id: 205bdce >2014-04-30 09:23:07,029 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-91) [3f17578] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 09:23:07,033 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 7dfaeb7b >2014-04-30 09:23:07,482 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-47) FINISH, ConnectStoragePoolVDSCommand, log id: 7dfaeb7b >2014-04-30 09:23:07,482 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-47) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 09:23:07,492 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-91) [3f349ac] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:23:07,493 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) [3f349ac] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 4b675ea5 >2014-04-30 09:23:07,495 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-91) [3f349ac] FINISH, SetVdsStatusVDSCommand, log id: 4b675ea5 >2014-04-30 09:23:07,501 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [3f349ac] Correlation ID: 3f349ac, Job ID: d8ac45fd-8135-4669-9d7a-a0d27ead6984, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 09:23:07,508 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [3f349ac] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 09:23:07,510 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-91) [3f349ac] Correlation ID: 6fad2fa1, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 09:23:07,513 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-91) [7729b5a4] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:23:07,514 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-91) [4cf6ef84] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:23:07,515 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-91) [4cf6ef84] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 09:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-72) [58cc4101] Autorecovering 1 hosts >2014-04-30 09:25:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-72) [58cc4101] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 09:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 09:25:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 09:25:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 4726258d >2014-04-30 09:25:00,014 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] FINISH, SetVdsStatusVDSCommand, log id: 4726258d >2014-04-30 09:25:00,015 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 5ae42196 >2014-04-30 09:25:00,481 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 6c8d6617 >2014-04-30 09:25:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] Failed in GetHardwareInfoVDS method >2014-04-30 09:25:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:25:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@140490ac >2014-04-30 09:25:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] HostName = 10.34.27.209 >2014-04-30 09:25:00,706 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:25:00,706 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-72) [264b137a] FINISH, GetHardwareInfoVDSCommand, log id: 6c8d6617 >2014-04-30 09:25:00,711 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-72) [264b137a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:25:00,722 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-72) [75644c74] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:25:00,723 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-72) [1e578a9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:25:00,724 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-72) [1e578a9] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 5ae42196 >2014-04-30 09:25:00,740 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [1e578a9] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 09:25:00,742 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-72) [1e578a9] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:25:01,864 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [764adf7a] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 4b80d8d0 >2014-04-30 09:25:02,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [764adf7a] Failed in GetHardwareInfoVDS method >2014-04-30 09:25:02,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [764adf7a] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:25:02,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [764adf7a] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@781a6669 >2014-04-30 09:25:02,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [764adf7a] HostName = 10.34.27.209 >2014-04-30 09:25:02,084 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [764adf7a] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:25:02,084 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-46) [764adf7a] FINISH, GetHardwareInfoVDSCommand, log id: 4b80d8d0 >2014-04-30 09:25:02,088 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [764adf7a] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:25:02,097 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [764adf7a] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 09:25:02,321 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-46) [4a4a709f] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:25:02,323 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-46) [69e0ef3f] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:25:02,329 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [69e0ef3f] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4cb7bc21 >2014-04-30 09:28:02,330 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [69e0ef3f] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 09:28:02,330 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-46) [69e0ef3f] FINISH, ConnectStorageServerVDSCommand, log id: 4cb7bc21 >2014-04-30 09:28:02,330 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-46) [69e0ef3f] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 09:28:02,334 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 15a5b67 >2014-04-30 09:28:02,777 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 15a5b67 >2014-04-30 09:28:02,778 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 09:28:02,789 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-46) [46b89a53] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:28:02,790 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [46b89a53] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 543dd9d0 >2014-04-30 09:28:02,792 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-46) [46b89a53] FINISH, SetVdsStatusVDSCommand, log id: 543dd9d0 >2014-04-30 09:28:02,800 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [46b89a53] Correlation ID: 46b89a53, Job ID: 1a29b95d-a9ad-4fb1-be5a-80464bbb78f6, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 09:28:02,809 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [46b89a53] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 09:28:02,811 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-46) [46b89a53] Correlation ID: 4a4a709f, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 09:28:02,817 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-46) [78a5551f] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:28:02,818 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-46) [775dfae9] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:28:02,819 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-46) [775dfae9] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 09:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-59) [2c46419] Autorecovering 1 hosts >2014-04-30 09:30:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-59) [2c46419] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 09:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 09:30:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 09:30:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: e235bf1 >2014-04-30 09:30:00,012 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] FINISH, SetVdsStatusVDSCommand, log id: e235bf1 >2014-04-30 09:30:00,014 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] START, ActivateVdsVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991), log id: 48ad68be >2014-04-30 09:30:00,494 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 481fde35 >2014-04-30 09:30:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] Failed in GetHardwareInfoVDS method >2014-04-30 09:30:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:30:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@413d785 >2014-04-30 09:30:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] HostName = 10.34.27.209 >2014-04-30 09:30:00,719 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:30:00,719 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] FINISH, GetHardwareInfoVDSCommand, log id: 481fde35 >2014-04-30 09:30:00,734 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-59) [79d3d5b5] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshHost(VdsManager.java:403) > at org.ovirt.engine.core.vdsbroker.VdsManager.activate(VdsManager.java:389) > at org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand.executeVdsIdCommand(ActivateVdsVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.VdsIdVDSCommandBase.executeVDSCommand(VdsIdVDSCommandBase.java:42) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.bll.VDSBrokerFrontendImpl.RunVdsCommand(VDSBrokerFrontendImpl.java:33) > at org.ovirt.engine.core.bll.CommandBase.runVdsCommand(CommandBase.java:1982) > at org.ovirt.engine.core.bll.ActivateVdsCommand.executeCommand(ActivateVdsCommand.java:53) > at org.ovirt.engine.core.bll.CommandBase.executeWithoutTransaction(CommandBase.java:1134) > at org.ovirt.engine.core.bll.CommandBase.executeActionInTransactionScope(CommandBase.java:1219) > at org.ovirt.engine.core.bll.CommandBase.runInTransaction(CommandBase.java:1895) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInSuppressed(TransactionSupport.java:174) > at org.ovirt.engine.core.utils.transaction.TransactionSupport.executeInScope(TransactionSupport.java:116) > at org.ovirt.engine.core.bll.CommandBase.execute(CommandBase.java:1239) > at org.ovirt.engine.core.bll.CommandBase.executeAction(CommandBase.java:362) > at org.ovirt.engine.core.bll.Backend.runAction(Backend.java:416) > at org.ovirt.engine.core.bll.Backend.runActionImpl(Backend.java:395) > at org.ovirt.engine.core.bll.Backend.runInternalAction(Backend.java:344) > at sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.jboss.as.ee.component.ManagedReferenceMethodInterceptorFactory$ManagedReferenceMethodInterceptor.processInvocation(ManagedReferenceMethodInterceptorFactory.java:72) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.WeavedInterceptor.processInvocation(WeavedInterceptor.java:53) > at org.jboss.as.ee.component.interceptors.UserInterceptorFactory$1.processInvocation(UserInterceptorFactory.java:58) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.invocationmetrics.ExecutionTimeInterceptor.processInvocation(ExecutionTimeInterceptor.java:43) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.InitialInterceptor.processInvocation(InitialInterceptor.java:21) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.interceptors.ComponentDispatcherInterceptor.processInvocation(ComponentDispatcherInterceptor.java:53) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:52) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.invokeInNoTx(CMTTxInterceptor.java:259) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.supports(CMTTxInterceptor.java:398) > at org.jboss.as.ejb3.tx.CMTTxInterceptor.processInvocation(CMTTxInterceptor.java:242) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.CurrentInvocationContextInterceptor.processInvocation(CurrentInvocationContextInterceptor.java:41) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.ShutDownInterceptorFactory$1.processInvocation(ShutDownInterceptorFactory.java:64) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ejb3.component.interceptors.LoggingInterceptor.processInvocation(LoggingInterceptor.java:59) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.NamespaceContextInterceptor.processInvocation(NamespaceContextInterceptor.java:50) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.as.ee.component.TCCLInterceptor.processInvocation(TCCLInterceptor.java:45) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ViewService$View.invoke(ViewService.java:165) > at org.jboss.as.ee.component.ViewDescription$1.processInvocation(ViewDescription.java:182) > at org.jboss.invocation.InterceptorContext.proceed(InterceptorContext.java:288) > at org.jboss.invocation.ChainedInterceptor.processInvocation(ChainedInterceptor.java:61) > at org.jboss.as.ee.component.ProxyInvocationHandler.invoke(ProxyInvocationHandler.java:72) > at org.ovirt.engine.core.bll.interfaces.BackendInternal$$$view8.runInternalAction(Unknown Source) > at org.ovirt.engine.core.bll.AutoRecoveryManager.check(AutoRecoveryManager.java:129) > at org.ovirt.engine.core.bll.AutoRecoveryManager.onTimer(AutoRecoveryManager.java:59) > at sun.reflect.GeneratedMethodAccessor91.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:30:00,750 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-59) [65aba71c] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:30:00,751 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-59) [676c50b4] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:30:00,752 INFO [org.ovirt.engine.core.vdsbroker.ActivateVdsVDSCommand] (DefaultQuartzScheduler_Worker-59) [676c50b4] FINISH, ActivateVdsVDSCommand, return: Host[10.34.27.209], log id: 48ad68be >2014-04-30 09:30:00,765 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [676c50b4] Activate finished. Lock released. Monitoring can run now for host 10.34.27.209 from data-center NFS >2014-04-30 09:30:00,767 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-59) [676c50b4] Lock freed to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:30:03,174 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [10535fa3] START, GetHardwareInfoVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, vds=Host[10.34.27.209]), log id: 46244afc >2014-04-30 09:30:03,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [10535fa3] Failed in GetHardwareInfoVDS method >2014-04-30 09:30:03,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [10535fa3] Error code hwInfoErr and error message VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:30:03,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [10535fa3] Command org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand return value > org.ovirt.engine.core.vdsbroker.vdsbroker.VDSInfoReturnForXmlRpc@78b70a5d >2014-04-30 09:30:03,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [10535fa3] HostName = 10.34.27.209 >2014-04-30 09:30:03,408 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [10535fa3] Command GetHardwareInfoVDS execution failed. Exception: VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information >2014-04-30 09:30:03,408 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand] (DefaultQuartzScheduler_Worker-80) [10535fa3] FINISH, GetHardwareInfoVDSCommand, log id: 46244afc >2014-04-30 09:30:03,413 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [10535fa3] Correlation ID: null, Call Stack: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSErrorException: VDSGenericException: VDSErrorException: Failed to GetHardwareInfoVDS, error = Failed to read hardware information > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.createDefaultConcreteException(VdsBrokerCommand.java:61) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.createException(BrokerCommandBase.java:199) > at org.ovirt.engine.core.vdsbroker.vdsbroker.BrokerCommandBase.proceedProxyReturnValue(BrokerCommandBase.java:185) > at org.ovirt.engine.core.vdsbroker.vdsbroker.GetHardwareInfoVDSCommand.executeVdsBrokerCommand(GetHardwareInfoVDSCommand.java:13) > at org.ovirt.engine.core.vdsbroker.vdsbroker.VdsBrokerCommand.executeVDSCommand(VdsBrokerCommand.java:96) > at org.ovirt.engine.core.vdsbroker.VDSCommandBase.executeCommand(VDSCommandBase.java:56) > at org.ovirt.engine.core.dal.VdcCommandBase.execute(VdcCommandBase.java:28) > at org.ovirt.engine.core.vdsbroker.ResourceManager.runVdsCommand(ResourceManager.java:397) > at org.ovirt.engine.core.vdsbroker.VdsManager.refreshCapabilities(VdsManager.java:548) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.refreshVdsRunTimeInfo(VdsUpdateRunTimeInfo.java:486) > at org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo.Refresh(VdsUpdateRunTimeInfo.java:334) > at org.ovirt.engine.core.vdsbroker.VdsManager.OnTimer(VdsManager.java:236) > at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.ovirt.engine.core.utils.timer.JobWrapper.execute(JobWrapper.java:60) > at org.quartz.core.JobRunShell.run(JobRunShell.java:213) > at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:557) >, Custom Event ID: -1, Message: Could not get hardware information for host 10.34.27.209 >2014-04-30 09:30:03,422 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [10535fa3] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: State was set to Up for host 10.34.27.209. >2014-04-30 09:30:03,662 INFO [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (DefaultQuartzScheduler_Worker-80) [4e8c3740] Running command: InitVdsOnUpCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:30:03,664 INFO [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [24e619b9] Running command: ConnectHostToStoragePoolServersCommand internal: true. Entities affected : ID: f03e56af-56f8-4150-9517-271bec2305d9 Type: StoragePool >2014-04-30 09:30:03,670 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [24e619b9] START, ConnectStorageServerVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, storageType = NFS, connectionList = [{ id: 8925e659-8b90-40f7-8e79-184a928c496f, connection: 10.34.1.251:/mnt/export2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 19363eb9-2267-4f2b-9c03-2035530f41d2, connection: 10.34.1.251:/mnt/storage2, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };{ id: 854c3686-561c-4079-af7a-2115dab8d0b5, connection: 10.34.27.209:/mnt/storage, iqn: null, vfsType: null, mountOptions: null, nfsVersion: null, nfsRetrans: null, nfsTimeo: null };]), log id: 4d9dd49d >2014-04-30 09:33:03,671 ERROR [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [24e619b9] Command ConnectStorageServerVDS execution failed. Exception: VDSNetworkException: java.util.concurrent.TimeoutException >2014-04-30 09:33:03,671 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStorageServerVDSCommand] (DefaultQuartzScheduler_Worker-80) [24e619b9] FINISH, ConnectStorageServerVDSCommand, log id: 4d9dd49d >2014-04-30 09:33:03,671 ERROR [org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand] (DefaultQuartzScheduler_Worker-80) [24e619b9] Command org.ovirt.engine.core.bll.storage.ConnectHostToStoragePoolServersCommand throw Vdc Bll exception. With error message VdcBLLException: org.ovirt.engine.core.vdsbroker.vdsbroker.VDSNetworkException: java.util.concurrent.TimeoutException (Failed with error VDS_NETWORK_ERROR and code 5022) >2014-04-30 09:33:03,675 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) START, ConnectStoragePoolVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, storagePoolId = f03e56af-56f8-4150-9517-271bec2305d9, vds_spm_id = 2, masterDomainId = b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion = 1), log id: 11a8bb62 >2014-04-30 09:33:04,113 INFO [org.ovirt.engine.core.vdsbroker.vdsbroker.ConnectStoragePoolVDSCommand] (pool-4-thread-50) FINISH, ConnectStoragePoolVDSCommand, log id: 11a8bb62 >2014-04-30 09:33:04,113 ERROR [org.ovirt.engine.core.bll.InitVdsOnUpCommand] (pool-4-thread-50) Could not connect host 10.34.27.209 to pool NFS with the message: Error storage pool connection: ('spUUID=f03e56af-56f8-4150-9517-271bec2305d9, msdUUID=b115b57b-7418-4b60-86b2-64af22fcd1db, masterVersion=1, hostID=2, scsiKey=f03e56af-56f8-4150-9517-271bec2305d9',) >2014-04-30 09:33:04,124 INFO [org.ovirt.engine.core.bll.SetNonOperationalVdsCommand] (DefaultQuartzScheduler_Worker-80) [6dac543] Running command: SetNonOperationalVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:33:04,125 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [6dac543] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=NonOperational, nonOperationalReason=STORAGE_DOMAIN_UNREACHABLE, stopSpmFailureLogged=false), log id: 112f774f >2014-04-30 09:33:04,127 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-80) [6dac543] FINISH, SetVdsStatusVDSCommand, log id: 112f774f >2014-04-30 09:33:04,135 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [6dac543] Correlation ID: 6dac543, Job ID: 477abfd5-a099-4682-806c-9e3018e94859, Call Stack: null, Custom Event ID: -1, Message: Host 10.34.27.209 cannot access one of the Storage Domains attached to the Data Center NFS. Setting Host state to Non-Operational. >2014-04-30 09:33:04,144 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [6dac543] Correlation ID: null, Call Stack: null, Custom Event ID: -1, Message: Failed to verify Power Management configuration for Host 10.34.27.209. >2014-04-30 09:33:04,146 INFO [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (DefaultQuartzScheduler_Worker-80) [6dac543] Correlation ID: 4e8c3740, Call Stack: null, Custom Event ID: -1, Message: Failed to connect Host 10.34.27.209 to Storage Pool NFS >2014-04-30 09:33:04,150 INFO [org.ovirt.engine.core.bll.HandleVdsCpuFlagsOrClusterChangedCommand] (DefaultQuartzScheduler_Worker-80) [1d03d0b6] Running command: HandleVdsCpuFlagsOrClusterChangedCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:33:04,151 INFO [org.ovirt.engine.core.bll.HandleVdsVersionCommand] (DefaultQuartzScheduler_Worker-80) [4a658a95] Running command: HandleVdsVersionCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:33:04,152 INFO [org.ovirt.engine.core.vdsbroker.VdsUpdateRunTimeInfo] (DefaultQuartzScheduler_Worker-80) [4a658a95] Host 53fd133a-9dc6-11e2-abd9-001a4a223991 : 10.34.27.209 is already in NonOperational status for reason STORAGE_DOMAIN_UNREACHABLE. SetNonOperationalVds command is skipped. >2014-04-30 09:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-41) [b910aff] Autorecovering 1 hosts >2014-04-30 09:35:00,004 INFO [org.ovirt.engine.core.bll.AutoRecoveryManager] (DefaultQuartzScheduler_Worker-41) [b910aff] Autorecovering hosts id: 53fd133a-9dc6-11e2-abd9-001a4a223991, name : 10.34.27.209 >2014-04-30 09:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [5a37dfa0] Lock Acquired to object EngineLock [exclusiveLocks= key: 53fd133a-9dc6-11e2-abd9-001a4a223991 value: VDS >, sharedLocks= ] >2014-04-30 09:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [5a37dfa0] Running command: ActivateVdsCommand internal: true. Entities affected : ID: 53fd133a-9dc6-11e2-abd9-001a4a223991 Type: VDS >2014-04-30 09:35:00,005 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [5a37dfa0] Before acquiring lock in order to prevent monitoring for host 10.34.27.209 from data-center NFS >2014-04-30 09:35:00,006 INFO [org.ovirt.engine.core.bll.ActivateVdsCommand] (DefaultQuartzScheduler_Worker-41) [5a37dfa0] Lock acquired, from now a monitoring of host will be skipped for host 10.34.27.209 from data-center NFS >2014-04-30 09:35:00,008 INFO [org.ovirt.engine.core.vdsbroker.SetVdsStatusVDSCommand] (DefaultQuartzScheduler_Worker-41) [5a37dfa0] START, SetVdsStatusVDSCommand(HostName = 10.34.27.209, HostId = 53fd133a-9dc6-11e2-abd9-001a4a223991, status=Unassigned, nonOperationalReason=NONE, stopSpmFailureLogged=false), log id: 7bfaf8c2 >